Allow common redis and leveldb connections (#12385)
* Allow common redis and leveldb connections Prevents multiple reopening of redis and leveldb connections to the same place by sharing connections. Further allows for more configurable redis connection type using the redisURI and a leveldbURI scheme. Signed-off-by: Andrew Thornton <art27@cantab.net> * add unit-test Signed-off-by: Andrew Thornton <art27@cantab.net> * as per @lunny Signed-off-by: Andrew Thornton <art27@cantab.net> * add test Signed-off-by: Andrew Thornton <art27@cantab.net> * Update modules/cache/cache_redis.go * Update modules/queue/queue_disk.go * Update modules/cache/cache_redis.go * Update modules/cache/cache_redis.go * Update modules/queue/unique_queue_disk.go * Update modules/queue/queue_disk.go * Update modules/queue/unique_queue_disk.go * Update modules/session/redis.go Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lauris BH <lauris@nix.lv>
This commit is contained in:
zeripath
@ -467,8 +467,10 @@ LENGTH = 20
|
||||
BATCH_LENGTH = 20
|
||||
; Connection string for redis queues this will store the redis connection string.
|
||||
CONN_STR = "addrs=127.0.0.1:6379 db=0"
|
||||
; Provide the suffix of the default redis queue name - specific queues can be overriden within in their [queue.name] sections.
|
||||
; Provides the suffix of the default redis/disk queue name - specific queues can be overriden within in their [queue.name] sections.
|
||||
QUEUE_NAME = "_queue"
|
||||
; Provides the suffix of the default redis/disk unique queue set name - specific queues can be overriden within in their [queue.name] sections.
|
||||
SET_NAME = "_unique"
|
||||
; If the queue cannot be created at startup - level queues may need a timeout at startup - wrap the queue:
|
||||
WRAP_IF_NECESSARY = true
|
||||
; Attempt to create the wrapped queue at max
|
||||
|
||||
@ -308,15 +308,13 @@ relation to port exhaustion.
|
||||
## Queue (`queue` and `queue.*`)
|
||||
|
||||
- `TYPE`: **persistable-channel**: General queue type, currently support: `persistable-channel`, `channel`, `level`, `redis`, `dummy`
|
||||
- `DATADIR`: **queues/**: Base DataDir for storing persistent and level queues. `DATADIR` for inidividual queues can be set in `queue.name` sections but will default to `DATADIR/`**`name`**.
|
||||
- `DATADIR`: **queues/**: Base DataDir for storing persistent and level queues. `DATADIR` for individual queues can be set in `queue.name` sections but will default to `DATADIR/`**`name`**.
|
||||
- `LENGTH`: **20**: Maximal queue size before channel queues block
|
||||
- `BATCH_LENGTH`: **20**: Batch data before passing to the handler
|
||||
- `CONN_STR`: **addrs=127.0.0.1:6379 db=0**: Connection string for the redis queue type.
|
||||
- `QUEUE_NAME`: **_queue**: The suffix for default redis queue name. Individual queues will default to **`name`**`QUEUE_NAME` but can be overriden in the specific `queue.name` section.
|
||||
- `SET_NAME`: **_unique**: The suffix that will added to the default redis
|
||||
set name for unique queues. Individual queues will default to
|
||||
**`name`**`QUEUE_NAME`_`SET_NAME`_ but can be overridden in the specific
|
||||
`queue.name` section.
|
||||
- `CONN_STR`: **redis://127.0.0.1:6379/0**: Connection string for the redis queue type. Options can be set using query params. Similarly LevelDB options can also be set using: **leveldb://relative/path?option=value** or **leveldb:///absolute/path?option=value**
|
||||
- `QUEUE_NAME`: **_queue**: The suffix for default redis and disk queue name. Individual queues will default to **`name`**`QUEUE_NAME` but can be overriden in the specific `queue.name` section.
|
||||
- `SET_NAME`: **_unique**: The suffix that will be added to the default redis and disk queue `set` name for unique queues. Individual queues will default to
|
||||
**`name`**`QUEUE_NAME`_`SET_NAME`_ but can be overridden in the specific `queue.name` section.
|
||||
- `WRAP_IF_NECESSARY`: **true**: Will wrap queues with a timeoutable queue if the selected queue is not ready to be created - (Only relevant for the level queue.)
|
||||
- `MAX_ATTEMPTS`: **10**: Maximum number of attempts to create the wrapped queue
|
||||
- `TIMEOUT`: **GRACEFUL_HAMMER_TIME + 30s**: Timeout the creation of the wrapped queue if it takes longer than this to create.
|
||||
@ -459,7 +457,7 @@ set name for unique queues. Individual queues will default to
|
||||
- `ADAPTER`: **memory**: Cache engine adapter, either `memory`, `redis`, or `memcache`.
|
||||
- `INTERVAL`: **60**: Garbage Collection interval (sec), for memory cache only.
|
||||
- `HOST`: **\<empty\>**: Connection string for `redis` and `memcache`.
|
||||
- Redis: `network=tcp,addr=127.0.0.1:6379,password=macaron,db=0,pool_size=100,idle_timeout=180`
|
||||
- Redis: `redis://:macaron@127.0.0.1:6379/0?pool_size=100&idle_timeout=180s`
|
||||
- Memcache: `127.0.0.1:9090;127.0.0.1:9091`
|
||||
- `ITEM_TTL`: **16h**: Time to keep items in cache if not used, Setting it to 0 disables caching.
|
||||
|
||||
@ -708,7 +706,7 @@ Task queue configuration has been moved to `queue.task`. However, the below conf
|
||||
|
||||
- `QUEUE_TYPE`: **channel**: Task queue type, could be `channel` or `redis`.
|
||||
- `QUEUE_LENGTH`: **1000**: Task queue length, available only when `QUEUE_TYPE` is `channel`.
|
||||
- `QUEUE_CONN_STR`: **addrs=127.0.0.1:6379 db=0**: Task queue connection string, available only when `QUEUE_TYPE` is `redis`. If redis needs a password, use `addrs=127.0.0.1:6379 password=123 db=0`.
|
||||
- `QUEUE_CONN_STR`: **redis://127.0.0.1:6379/0**: Task queue connection string, available only when `QUEUE_TYPE` is `redis`. If redis needs a password, use `redis://123@127.0.0.1:6379/0`.
|
||||
|
||||
## Migrations (`migrations`)
|
||||
|
||||
|
||||
3
go.mod
3
go.mod
@ -38,7 +38,7 @@ require (
|
||||
github.com/go-enry/go-enry/v2 v2.5.2
|
||||
github.com/go-git/go-billy/v5 v5.0.0
|
||||
github.com/go-git/go-git/v5 v5.1.0
|
||||
github.com/go-redis/redis v6.15.2+incompatible
|
||||
github.com/go-redis/redis/v7 v7.4.0
|
||||
github.com/go-sql-driver/mysql v1.5.0
|
||||
github.com/go-swagger/go-swagger v0.25.0
|
||||
github.com/go-testfixtures/testfixtures/v3 v3.4.0
|
||||
@ -88,6 +88,7 @@ require (
|
||||
github.com/shurcooL/httpfs v0.0.0-20190527155220-6a4d4a70508b // indirect
|
||||
github.com/shurcooL/vfsgen v0.0.0-20181202132449-6a9ea43bcacd
|
||||
github.com/stretchr/testify v1.6.1
|
||||
github.com/syndtr/goleveldb v1.0.0
|
||||
github.com/tecbot/gorocksdb v0.0.0-20181010114359-8752a9433481 // indirect
|
||||
github.com/tinylib/msgp v1.1.2 // indirect
|
||||
github.com/tstranex/u2f v1.0.0
|
||||
|
||||
7
go.sum
7
go.sum
@ -342,6 +342,8 @@ github.com/go-openapi/validate v0.19.10 h1:tG3SZ5DC5KF4cyt7nqLVcQXGj5A7mpaYkAcNP
|
||||
github.com/go-openapi/validate v0.19.10/go.mod h1:RKEZTUWDkxKQxN2jDT7ZnZi2bhZlbNMAuKvKB+IaGx8=
|
||||
github.com/go-redis/redis v6.15.2+incompatible h1:9SpNVG76gr6InJGxoZ6IuuxaCOQwDAhzyXg+Bs+0Sb4=
|
||||
github.com/go-redis/redis v6.15.2+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA=
|
||||
github.com/go-redis/redis/v7 v7.4.0 h1:7obg6wUoj05T0EpY0o8B59S9w5yeMWql7sw2kwNW1x4=
|
||||
github.com/go-redis/redis/v7 v7.4.0/go.mod h1:JDNMw23GTyLNC4GZu9njt15ctBQVn7xjRfnwdHj/Dcg=
|
||||
github.com/go-sql-driver/mysql v1.4.1 h1:g24URVg0OFbNUTx9qqY1IRZ9D9z3iPyi5zKhQZpNwpA=
|
||||
github.com/go-sql-driver/mysql v1.4.1/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
|
||||
github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gGcHOs=
|
||||
@ -730,9 +732,13 @@ github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W
|
||||
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w=
|
||||
github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.10.1 h1:q/mM8GF/n0shIN8SaAZ0V+jnLPzen6WIVZdiwrRlMlo=
|
||||
github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo=
|
||||
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/onsi/gomega v1.7.0 h1:XPnZz8VVBHjVsy1vzJmRwIcSwiUO+JFfrv/xGiigmME=
|
||||
github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
|
||||
github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=
|
||||
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
|
||||
@ -1014,6 +1020,7 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL
|
||||
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
|
||||
1
modules/cache/cache.go
vendored
1
modules/cache/cache.go
vendored
@ -13,7 +13,6 @@ import (
|
||||
mc "gitea.com/macaron/cache"
|
||||
|
||||
_ "gitea.com/macaron/cache/memcache" // memcache plugin for cache
|
||||
_ "gitea.com/macaron/cache/redis"
|
||||
)
|
||||
|
||||
var (
|
||||
|
||||
@ -1,35 +1,23 @@
|
||||
// Copyright 2013 Beego Authors
|
||||
// Copyright 2014 The Macaron Authors
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License"): you may
|
||||
// not use this file except in compliance with the License. You may obtain
|
||||
// a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
// License for the specific language governing permissions and limitations
|
||||
// under the License.
|
||||
// Copyright 2020 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package cache
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/go-redis/redis"
|
||||
"github.com/unknwon/com"
|
||||
"gopkg.in/ini.v1"
|
||||
"code.gitea.io/gitea/modules/nosql"
|
||||
|
||||
"gitea.com/macaron/cache"
|
||||
"github.com/go-redis/redis/v7"
|
||||
"github.com/unknwon/com"
|
||||
)
|
||||
|
||||
// RedisCacher represents a redis cache adapter implementation.
|
||||
type RedisCacher struct {
|
||||
c *redis.Client
|
||||
c redis.UniversalClient
|
||||
prefix string
|
||||
hsetName string
|
||||
occupyMode bool
|
||||
@ -112,7 +100,7 @@ func (c *RedisCacher) IsExist(key string) bool {
|
||||
// Flush deletes all cached data.
|
||||
func (c *RedisCacher) Flush() error {
|
||||
if c.occupyMode {
|
||||
return c.c.FlushDb().Err()
|
||||
return c.c.FlushDB().Err()
|
||||
}
|
||||
|
||||
keys, err := c.c.HKeys(c.hsetName).Result()
|
||||
@ -131,46 +119,20 @@ func (c *RedisCacher) StartAndGC(opts cache.Options) error {
|
||||
c.hsetName = "MacaronCache"
|
||||
c.occupyMode = opts.OccupyMode
|
||||
|
||||
cfg, err := ini.Load([]byte(strings.Replace(opts.AdapterConfig, ",", "\n", -1)))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
uri := nosql.ToRedisURI(opts.AdapterConfig)
|
||||
|
||||
opt := &redis.Options{
|
||||
Network: "tcp",
|
||||
}
|
||||
for k, v := range cfg.Section("").KeysHash() {
|
||||
c.c = nosql.GetManager().GetRedisClient(uri.String())
|
||||
|
||||
for k, v := range uri.Query() {
|
||||
switch k {
|
||||
case "network":
|
||||
opt.Network = v
|
||||
case "addr":
|
||||
opt.Addr = v
|
||||
case "password":
|
||||
opt.Password = v
|
||||
case "db":
|
||||
opt.DB = com.StrTo(v).MustInt()
|
||||
case "pool_size":
|
||||
opt.PoolSize = com.StrTo(v).MustInt()
|
||||
case "idle_timeout":
|
||||
opt.IdleTimeout, err = time.ParseDuration(v + "s")
|
||||
if err != nil {
|
||||
return fmt.Errorf("error parsing idle timeout: %v", err)
|
||||
}
|
||||
case "hset_name":
|
||||
c.hsetName = v
|
||||
c.hsetName = v[0]
|
||||
case "prefix":
|
||||
c.prefix = v
|
||||
default:
|
||||
return fmt.Errorf("session/redis: unsupported option '%s'", k)
|
||||
c.prefix = v[0]
|
||||
}
|
||||
}
|
||||
|
||||
c.c = redis.NewClient(opt)
|
||||
if err = c.c.Ping().Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
return c.c.Ping().Err()
|
||||
}
|
||||
|
||||
func init() {
|
||||
25
modules/nosql/leveldb.go
Normal file
25
modules/nosql/leveldb.go
Normal file
@ -0,0 +1,25 @@
|
||||
// Copyright 2020 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package nosql
|
||||
|
||||
import "net/url"
|
||||
|
||||
// ToLevelDBURI converts old style connections to a LevelDBURI
|
||||
//
|
||||
// A LevelDBURI matches the pattern:
|
||||
//
|
||||
// leveldb://path[?[option=value]*]
|
||||
//
|
||||
// We have previously just provided the path but this prevent other options
|
||||
func ToLevelDBURI(connection string) *url.URL {
|
||||
uri, err := url.Parse(connection)
|
||||
if err == nil && uri.Scheme == "leveldb" {
|
||||
return uri
|
||||
}
|
||||
uri, _ = url.Parse("leveldb://common")
|
||||
uri.Host = ""
|
||||
uri.Path = connection
|
||||
return uri
|
||||
}
|
||||
71
modules/nosql/manager.go
Normal file
71
modules/nosql/manager.go
Normal file
@ -0,0 +1,71 @@
|
||||
// Copyright 2020 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package nosql
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/go-redis/redis/v7"
|
||||
"github.com/syndtr/goleveldb/leveldb"
|
||||
)
|
||||
|
||||
var manager *Manager
|
||||
|
||||
// Manager is the nosql connection manager
|
||||
type Manager struct {
|
||||
mutex sync.Mutex
|
||||
|
||||
RedisConnections map[string]*redisClientHolder
|
||||
LevelDBConnections map[string]*levelDBHolder
|
||||
}
|
||||
|
||||
type redisClientHolder struct {
|
||||
redis.UniversalClient
|
||||
name []string
|
||||
count int64
|
||||
}
|
||||
|
||||
func (r *redisClientHolder) Close() error {
|
||||
return manager.CloseRedisClient(r.name[0])
|
||||
}
|
||||
|
||||
type levelDBHolder struct {
|
||||
name []string
|
||||
count int64
|
||||
db *leveldb.DB
|
||||
}
|
||||
|
||||
func init() {
|
||||
_ = GetManager()
|
||||
}
|
||||
|
||||
// GetManager returns a Manager and initializes one as singleton is there's none yet
|
||||
func GetManager() *Manager {
|
||||
if manager == nil {
|
||||
manager = &Manager{
|
||||
RedisConnections: make(map[string]*redisClientHolder),
|
||||
LevelDBConnections: make(map[string]*levelDBHolder),
|
||||
}
|
||||
}
|
||||
return manager
|
||||
}
|
||||
|
||||
func valToTimeDuration(vs []string) (result time.Duration) {
|
||||
var err error
|
||||
for _, v := range vs {
|
||||
result, err = time.ParseDuration(v)
|
||||
if err != nil {
|
||||
var val int
|
||||
val, err = strconv.Atoi(v)
|
||||
result = time.Duration(val)
|
||||
}
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
151
modules/nosql/manager_leveldb.go
Normal file
151
modules/nosql/manager_leveldb.go
Normal file
@ -0,0 +1,151 @@
|
||||
// Copyright 2020 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package nosql
|
||||
|
||||
import (
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/syndtr/goleveldb/leveldb"
|
||||
"github.com/syndtr/goleveldb/leveldb/errors"
|
||||
"github.com/syndtr/goleveldb/leveldb/opt"
|
||||
)
|
||||
|
||||
// CloseLevelDB closes a levelDB
|
||||
func (m *Manager) CloseLevelDB(connection string) error {
|
||||
m.mutex.Lock()
|
||||
defer m.mutex.Unlock()
|
||||
db, ok := m.LevelDBConnections[connection]
|
||||
if !ok {
|
||||
connection = ToLevelDBURI(connection).String()
|
||||
db, ok = m.LevelDBConnections[connection]
|
||||
}
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
db.count--
|
||||
if db.count > 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, name := range db.name {
|
||||
delete(m.LevelDBConnections, name)
|
||||
}
|
||||
return db.db.Close()
|
||||
}
|
||||
|
||||
// GetLevelDB gets a levelDB for a particular connection
|
||||
func (m *Manager) GetLevelDB(connection string) (*leveldb.DB, error) {
|
||||
m.mutex.Lock()
|
||||
defer m.mutex.Unlock()
|
||||
db, ok := m.LevelDBConnections[connection]
|
||||
if ok {
|
||||
db.count++
|
||||
|
||||
return db.db, nil
|
||||
}
|
||||
dataDir := connection
|
||||
uri := ToLevelDBURI(connection)
|
||||
db = &levelDBHolder{
|
||||
name: []string{connection, uri.String()},
|
||||
}
|
||||
|
||||
dataDir = path.Join(uri.Host, uri.Path)
|
||||
opts := &opt.Options{}
|
||||
for k, v := range uri.Query() {
|
||||
switch replacer.Replace(strings.ToLower(k)) {
|
||||
case "blockcachecapacity":
|
||||
opts.BlockCacheCapacity, _ = strconv.Atoi(v[0])
|
||||
case "blockcacheevictremoved":
|
||||
opts.BlockCacheEvictRemoved, _ = strconv.ParseBool(v[0])
|
||||
case "blockrestartinterval":
|
||||
opts.BlockRestartInterval, _ = strconv.Atoi(v[0])
|
||||
case "blocksize":
|
||||
opts.BlockSize, _ = strconv.Atoi(v[0])
|
||||
case "compactionexpandlimitfactor":
|
||||
opts.CompactionExpandLimitFactor, _ = strconv.Atoi(v[0])
|
||||
case "compactiongpoverlapsfactor":
|
||||
opts.CompactionGPOverlapsFactor, _ = strconv.Atoi(v[0])
|
||||
case "compactionl0trigger":
|
||||
opts.CompactionL0Trigger, _ = strconv.Atoi(v[0])
|
||||
case "compactionsourcelimitfactor":
|
||||
opts.CompactionSourceLimitFactor, _ = strconv.Atoi(v[0])
|
||||
case "compactiontablesize":
|
||||
opts.CompactionTableSize, _ = strconv.Atoi(v[0])
|
||||
case "compactiontablesizemultiplier":
|
||||
opts.CompactionTableSizeMultiplier, _ = strconv.ParseFloat(v[0], 64)
|
||||
case "compactiontablesizemultiplierperlevel":
|
||||
for _, val := range v {
|
||||
f, _ := strconv.ParseFloat(val, 64)
|
||||
opts.CompactionTableSizeMultiplierPerLevel = append(opts.CompactionTableSizeMultiplierPerLevel, f)
|
||||
}
|
||||
case "compactiontotalsize":
|
||||
opts.CompactionTotalSize, _ = strconv.Atoi(v[0])
|
||||
case "compactiontotalsizemultiplier":
|
||||
opts.CompactionTotalSizeMultiplier, _ = strconv.ParseFloat(v[0], 64)
|
||||
case "compactiontotalsizemultiplierperlevel":
|
||||
for _, val := range v {
|
||||
f, _ := strconv.ParseFloat(val, 64)
|
||||
opts.CompactionTotalSizeMultiplierPerLevel = append(opts.CompactionTotalSizeMultiplierPerLevel, f)
|
||||
}
|
||||
case "compression":
|
||||
val, _ := strconv.Atoi(v[0])
|
||||
opts.Compression = opt.Compression(val)
|
||||
case "disablebufferpool":
|
||||
opts.DisableBufferPool, _ = strconv.ParseBool(v[0])
|
||||
case "disableblockcache":
|
||||
opts.DisableBlockCache, _ = strconv.ParseBool(v[0])
|
||||
case "disablecompactionbackoff":
|
||||
opts.DisableCompactionBackoff, _ = strconv.ParseBool(v[0])
|
||||
case "disablelargebatchtransaction":
|
||||
opts.DisableLargeBatchTransaction, _ = strconv.ParseBool(v[0])
|
||||
case "errorifexist":
|
||||
opts.ErrorIfExist, _ = strconv.ParseBool(v[0])
|
||||
case "errorifmissing":
|
||||
opts.ErrorIfMissing, _ = strconv.ParseBool(v[0])
|
||||
case "iteratorsamplingrate":
|
||||
opts.IteratorSamplingRate, _ = strconv.Atoi(v[0])
|
||||
case "nosync":
|
||||
opts.NoSync, _ = strconv.ParseBool(v[0])
|
||||
case "nowritemerge":
|
||||
opts.NoWriteMerge, _ = strconv.ParseBool(v[0])
|
||||
case "openfilescachecapacity":
|
||||
opts.OpenFilesCacheCapacity, _ = strconv.Atoi(v[0])
|
||||
case "readonly":
|
||||
opts.ReadOnly, _ = strconv.ParseBool(v[0])
|
||||
case "strict":
|
||||
val, _ := strconv.Atoi(v[0])
|
||||
opts.Strict = opt.Strict(val)
|
||||
case "writebuffer":
|
||||
opts.WriteBuffer, _ = strconv.Atoi(v[0])
|
||||
case "writel0pausetrigger":
|
||||
opts.WriteL0PauseTrigger, _ = strconv.Atoi(v[0])
|
||||
case "writel0slowdowntrigger":
|
||||
opts.WriteL0SlowdownTrigger, _ = strconv.Atoi(v[0])
|
||||
case "clientname":
|
||||
db.name = append(db.name, v[0])
|
||||
}
|
||||
}
|
||||
|
||||
var err error
|
||||
db.db, err = leveldb.OpenFile(dataDir, opts)
|
||||
if err != nil {
|
||||
if !errors.IsCorrupted(err) {
|
||||
return nil, err
|
||||
}
|
||||
db.db, err = leveldb.RecoverFile(dataDir, opts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
for _, name := range db.name {
|
||||
m.LevelDBConnections[name] = db
|
||||
}
|
||||
db.count++
|
||||
return db.db, nil
|
||||
}
|
||||
205
modules/nosql/manager_redis.go
Normal file
205
modules/nosql/manager_redis.go
Normal file
@ -0,0 +1,205 @@
|
||||
// Copyright 2020 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package nosql
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/go-redis/redis/v7"
|
||||
)
|
||||
|
||||
var replacer = strings.NewReplacer("_", "", "-", "")
|
||||
|
||||
// CloseRedisClient closes a redis client
|
||||
func (m *Manager) CloseRedisClient(connection string) error {
|
||||
m.mutex.Lock()
|
||||
defer m.mutex.Unlock()
|
||||
client, ok := m.RedisConnections[connection]
|
||||
if !ok {
|
||||
connection = ToRedisURI(connection).String()
|
||||
client, ok = m.RedisConnections[connection]
|
||||
}
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
client.count--
|
||||
if client.count > 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, name := range client.name {
|
||||
delete(m.RedisConnections, name)
|
||||
}
|
||||
return client.UniversalClient.Close()
|
||||
}
|
||||
|
||||
// GetRedisClient gets a redis client for a particular connection
|
||||
func (m *Manager) GetRedisClient(connection string) redis.UniversalClient {
|
||||
m.mutex.Lock()
|
||||
defer m.mutex.Unlock()
|
||||
client, ok := m.RedisConnections[connection]
|
||||
if ok {
|
||||
client.count++
|
||||
return client
|
||||
}
|
||||
|
||||
uri := ToRedisURI(connection)
|
||||
client, ok = m.RedisConnections[uri.String()]
|
||||
if ok {
|
||||
client.count++
|
||||
return client
|
||||
}
|
||||
client = &redisClientHolder{
|
||||
name: []string{connection, uri.String()},
|
||||
}
|
||||
|
||||
opts := &redis.UniversalOptions{}
|
||||
tlsConfig := &tls.Config{}
|
||||
|
||||
// Handle username/password
|
||||
if password, ok := uri.User.Password(); ok {
|
||||
opts.Password = password
|
||||
// Username does not appear to be handled by redis.Options
|
||||
opts.Username = uri.User.Username()
|
||||
} else if uri.User.Username() != "" {
|
||||
// assume this is the password
|
||||
opts.Password = uri.User.Username()
|
||||
}
|
||||
|
||||
// Now handle the uri query sets
|
||||
for k, v := range uri.Query() {
|
||||
switch replacer.Replace(strings.ToLower(k)) {
|
||||
case "addr":
|
||||
opts.Addrs = append(opts.Addrs, v...)
|
||||
case "addrs":
|
||||
opts.Addrs = append(opts.Addrs, strings.Split(v[0], ",")...)
|
||||
case "username":
|
||||
opts.Username = v[0]
|
||||
case "password":
|
||||
opts.Password = v[0]
|
||||
case "database":
|
||||
fallthrough
|
||||
case "db":
|
||||
opts.DB, _ = strconv.Atoi(v[0])
|
||||
case "maxretries":
|
||||
opts.MaxRetries, _ = strconv.Atoi(v[0])
|
||||
case "minretrybackoff":
|
||||
opts.MinRetryBackoff = valToTimeDuration(v)
|
||||
case "maxretrybackoff":
|
||||
opts.MaxRetryBackoff = valToTimeDuration(v)
|
||||
case "timeout":
|
||||
timeout := valToTimeDuration(v)
|
||||
if timeout != 0 {
|
||||
if opts.DialTimeout == 0 {
|
||||
opts.DialTimeout = timeout
|
||||
}
|
||||
if opts.ReadTimeout == 0 {
|
||||
opts.ReadTimeout = timeout
|
||||
}
|
||||
}
|
||||
case "dialtimeout":
|
||||
opts.DialTimeout = valToTimeDuration(v)
|
||||
case "readtimeout":
|
||||
opts.ReadTimeout = valToTimeDuration(v)
|
||||
case "writetimeout":
|
||||
opts.WriteTimeout = valToTimeDuration(v)
|
||||
case "poolsize":
|
||||
opts.PoolSize, _ = strconv.Atoi(v[0])
|
||||
case "minidleconns":
|
||||
opts.MinIdleConns, _ = strconv.Atoi(v[0])
|
||||
case "pooltimeout":
|
||||
opts.PoolTimeout = valToTimeDuration(v)
|
||||
case "idletimeout":
|
||||
opts.IdleTimeout = valToTimeDuration(v)
|
||||
case "idlecheckfrequency":
|
||||
opts.IdleCheckFrequency = valToTimeDuration(v)
|
||||
case "maxredirects":
|
||||
opts.MaxRedirects, _ = strconv.Atoi(v[0])
|
||||
case "readonly":
|
||||
opts.ReadOnly, _ = strconv.ParseBool(v[0])
|
||||
case "routebylatency":
|
||||
opts.RouteByLatency, _ = strconv.ParseBool(v[0])
|
||||
case "routerandomly":
|
||||
opts.RouteRandomly, _ = strconv.ParseBool(v[0])
|
||||
case "sentinelmasterid":
|
||||
fallthrough
|
||||
case "mastername":
|
||||
opts.MasterName = v[0]
|
||||
case "skipverify":
|
||||
fallthrough
|
||||
case "insecureskipverify":
|
||||
insecureSkipVerify, _ := strconv.ParseBool(v[0])
|
||||
tlsConfig.InsecureSkipVerify = insecureSkipVerify
|
||||
case "clientname":
|
||||
client.name = append(client.name, v[0])
|
||||
}
|
||||
}
|
||||
|
||||
switch uri.Scheme {
|
||||
case "redis+sentinels":
|
||||
fallthrough
|
||||
case "rediss+sentinel":
|
||||
opts.TLSConfig = tlsConfig
|
||||
fallthrough
|
||||
case "redis+sentinel":
|
||||
if uri.Host != "" {
|
||||
opts.Addrs = append(opts.Addrs, strings.Split(uri.Host, ",")...)
|
||||
}
|
||||
if uri.Path != "" {
|
||||
if db, err := strconv.Atoi(uri.Path); err == nil {
|
||||
opts.DB = db
|
||||
}
|
||||
}
|
||||
|
||||
client.UniversalClient = redis.NewFailoverClient(opts.Failover())
|
||||
case "redis+clusters":
|
||||
fallthrough
|
||||
case "rediss+cluster":
|
||||
opts.TLSConfig = tlsConfig
|
||||
fallthrough
|
||||
case "redis+cluster":
|
||||
if uri.Host != "" {
|
||||
opts.Addrs = append(opts.Addrs, strings.Split(uri.Host, ",")...)
|
||||
}
|
||||
if uri.Path != "" {
|
||||
if db, err := strconv.Atoi(uri.Path); err == nil {
|
||||
opts.DB = db
|
||||
}
|
||||
}
|
||||
client.UniversalClient = redis.NewClusterClient(opts.Cluster())
|
||||
case "redis+socket":
|
||||
simpleOpts := opts.Simple()
|
||||
simpleOpts.Network = "unix"
|
||||
simpleOpts.Addr = path.Join(uri.Host, uri.Path)
|
||||
client.UniversalClient = redis.NewClient(simpleOpts)
|
||||
case "rediss":
|
||||
opts.TLSConfig = tlsConfig
|
||||
fallthrough
|
||||
case "redis":
|
||||
if uri.Host != "" {
|
||||
opts.Addrs = append(opts.Addrs, strings.Split(uri.Host, ",")...)
|
||||
}
|
||||
if uri.Path != "" {
|
||||
if db, err := strconv.Atoi(uri.Path); err == nil {
|
||||
opts.DB = db
|
||||
}
|
||||
}
|
||||
client.UniversalClient = redis.NewClient(opts.Simple())
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, name := range client.name {
|
||||
m.RedisConnections[name] = client
|
||||
}
|
||||
|
||||
client.count++
|
||||
|
||||
return client
|
||||
}
|
||||
102
modules/nosql/redis.go
Normal file
102
modules/nosql/redis.go
Normal file
@ -0,0 +1,102 @@
|
||||
// Copyright 2020 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package nosql
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// The file contains common redis connection functions
|
||||
|
||||
// ToRedisURI converts old style connections to a RedisURI
|
||||
//
|
||||
// A RedisURI matches the pattern:
|
||||
//
|
||||
// redis://[username:password@]host[:port][/database][?[option=value]*]
|
||||
// rediss://[username:password@]host[:port][/database][?[option=value]*]
|
||||
// redis+socket://[username:password@]path[/database][?[option=value]*]
|
||||
// redis+sentinel://[password@]host1 [: port1][, host2 [:port2]][, hostN [:portN]][/ database][?[option=value]*]
|
||||
// redis+cluster://[password@]host1 [: port1][, host2 [:port2]][, hostN [:portN]][/ database][?[option=value]*]
|
||||
//
|
||||
// We have previously used a URI like:
|
||||
// addrs=127.0.0.1:6379 db=0
|
||||
// network=tcp,addr=127.0.0.1:6379,password=macaron,db=0,pool_size=100,idle_timeout=180
|
||||
//
|
||||
// We need to convert this old style to the new style
|
||||
func ToRedisURI(connection string) *url.URL {
|
||||
uri, err := url.Parse(connection)
|
||||
if err == nil && strings.HasPrefix(uri.Scheme, "redis") {
|
||||
// OK we're going to assume that this is a reasonable redis URI
|
||||
return uri
|
||||
}
|
||||
|
||||
// Let's set a nice default
|
||||
uri, _ = url.Parse("redis://127.0.0.1:6379/0")
|
||||
network := "tcp"
|
||||
query := uri.Query()
|
||||
|
||||
// OK so there are two types: Space delimited and Comma delimited
|
||||
// Let's assume that we have a space delimited string - as this is the most common
|
||||
fields := strings.Fields(connection)
|
||||
if len(fields) == 1 {
|
||||
// It's a comma delimited string, then...
|
||||
fields = strings.Split(connection, ",")
|
||||
|
||||
}
|
||||
for _, f := range fields {
|
||||
items := strings.SplitN(f, "=", 2)
|
||||
if len(items) < 2 {
|
||||
continue
|
||||
}
|
||||
switch strings.ToLower(items[0]) {
|
||||
case "network":
|
||||
if items[1] == "unix" {
|
||||
uri.Scheme = "redis+socket"
|
||||
}
|
||||
network = items[1]
|
||||
case "addrs":
|
||||
uri.Host = items[1]
|
||||
// now we need to handle the clustering
|
||||
if strings.Contains(items[1], ",") && network == "tcp" {
|
||||
uri.Scheme = "redis+cluster"
|
||||
}
|
||||
case "addr":
|
||||
uri.Host = items[1]
|
||||
case "password":
|
||||
uri.User = url.UserPassword(uri.User.Username(), items[1])
|
||||
case "username":
|
||||
password, set := uri.User.Password()
|
||||
if !set {
|
||||
uri.User = url.User(items[1])
|
||||
} else {
|
||||
uri.User = url.UserPassword(items[1], password)
|
||||
}
|
||||
case "db":
|
||||
uri.Path = "/" + items[1]
|
||||
case "idle_timeout":
|
||||
_, err := strconv.Atoi(items[1])
|
||||
if err == nil {
|
||||
query.Add("idle_timeout", items[1]+"s")
|
||||
} else {
|
||||
query.Add("idle_timeout", items[1])
|
||||
}
|
||||
default:
|
||||
// Other options become query params
|
||||
query.Add(items[0], items[1])
|
||||
}
|
||||
}
|
||||
|
||||
// Finally we need to fix up the Host if we have a unix port
|
||||
if uri.Scheme == "redis+socket" {
|
||||
query.Set("db", uri.Path)
|
||||
uri.Path = uri.Host
|
||||
uri.Host = ""
|
||||
}
|
||||
uri.RawQuery = query.Encode()
|
||||
|
||||
return uri
|
||||
}
|
||||
35
modules/nosql/redis_test.go
Normal file
35
modules/nosql/redis_test.go
Normal file
@ -0,0 +1,35 @@
|
||||
// Copyright 2020 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package nosql
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestToRedisURI(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
connection string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "old_default",
|
||||
connection: "addrs=127.0.0.1:6379 db=0",
|
||||
want: "redis://127.0.0.1:6379/0",
|
||||
},
|
||||
{
|
||||
name: "old_macaron_session_default",
|
||||
connection: "network=tcp,addr=127.0.0.1:6379,password=macaron,db=0,pool_size=100,idle_timeout=180",
|
||||
want: "redis://:macaron@127.0.0.1:6379/0?idle_timeout=180s&pool_size=100",
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := ToRedisURI(tt.connection); got == nil || got.String() != tt.want {
|
||||
t.Errorf(`ToRedisURI(%q) = %s, want %s`, tt.connection, got.String(), tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@ -5,6 +5,8 @@
|
||||
package queue
|
||||
|
||||
import (
|
||||
"code.gitea.io/gitea/modules/nosql"
|
||||
|
||||
"gitea.com/lunny/levelqueue"
|
||||
)
|
||||
|
||||
@ -15,6 +17,8 @@ const LevelQueueType Type = "level"
|
||||
type LevelQueueConfiguration struct {
|
||||
ByteFIFOQueueConfiguration
|
||||
DataDir string
|
||||
ConnectionString string
|
||||
QueueName string
|
||||
}
|
||||
|
||||
// LevelQueue implements a disk library queue
|
||||
@ -30,7 +34,11 @@ func NewLevelQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)
|
||||
}
|
||||
config := configInterface.(LevelQueueConfiguration)
|
||||
|
||||
byteFIFO, err := NewLevelQueueByteFIFO(config.DataDir)
|
||||
if len(config.ConnectionString) == 0 {
|
||||
config.ConnectionString = config.DataDir
|
||||
}
|
||||
|
||||
byteFIFO, err := NewLevelQueueByteFIFO(config.ConnectionString, config.QueueName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@ -52,16 +60,23 @@ var _ (ByteFIFO) = &LevelQueueByteFIFO{}
|
||||
// LevelQueueByteFIFO represents a ByteFIFO formed from a LevelQueue
|
||||
type LevelQueueByteFIFO struct {
|
||||
internal *levelqueue.Queue
|
||||
connection string
|
||||
}
|
||||
|
||||
// NewLevelQueueByteFIFO creates a ByteFIFO formed from a LevelQueue
|
||||
func NewLevelQueueByteFIFO(dataDir string) (*LevelQueueByteFIFO, error) {
|
||||
internal, err := levelqueue.Open(dataDir)
|
||||
func NewLevelQueueByteFIFO(connection, prefix string) (*LevelQueueByteFIFO, error) {
|
||||
db, err := nosql.GetManager().GetLevelDB(connection)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
internal, err := levelqueue.NewQueue(db, []byte(prefix), false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &LevelQueueByteFIFO{
|
||||
connection: connection,
|
||||
internal: internal,
|
||||
}, nil
|
||||
}
|
||||
@ -87,7 +102,9 @@ func (fifo *LevelQueueByteFIFO) Pop() ([]byte, error) {
|
||||
|
||||
// Close this fifo
|
||||
func (fifo *LevelQueueByteFIFO) Close() error {
|
||||
return fifo.internal.Close()
|
||||
err := fifo.internal.Close()
|
||||
_ = nosql.GetManager().CloseLevelDB(fifo.connection)
|
||||
return err
|
||||
}
|
||||
|
||||
// Len returns the length of the fifo
|
||||
|
||||
@ -5,12 +5,10 @@
|
||||
package queue
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
|
||||
"code.gitea.io/gitea/modules/log"
|
||||
"code.gitea.io/gitea/modules/nosql"
|
||||
|
||||
"github.com/go-redis/redis"
|
||||
"github.com/go-redis/redis/v7"
|
||||
)
|
||||
|
||||
// RedisQueueType is the type for redis queue
|
||||
@ -75,10 +73,7 @@ type RedisByteFIFO struct {
|
||||
|
||||
// RedisByteFIFOConfiguration is the configuration for the RedisByteFIFO
|
||||
type RedisByteFIFOConfiguration struct {
|
||||
Network string
|
||||
Addresses string
|
||||
Password string
|
||||
DBIndex int
|
||||
ConnectionString string
|
||||
QueueName string
|
||||
}
|
||||
|
||||
@ -87,21 +82,7 @@ func NewRedisByteFIFO(config RedisByteFIFOConfiguration) (*RedisByteFIFO, error)
|
||||
fifo := &RedisByteFIFO{
|
||||
queueName: config.QueueName,
|
||||
}
|
||||
dbs := strings.Split(config.Addresses, ",")
|
||||
if len(dbs) == 0 {
|
||||
return nil, errors.New("no redis host specified")
|
||||
} else if len(dbs) == 1 {
|
||||
fifo.client = redis.NewClient(&redis.Options{
|
||||
Network: config.Network,
|
||||
Addr: strings.TrimSpace(dbs[0]), // use default Addr
|
||||
Password: config.Password, // no password set
|
||||
DB: config.DBIndex, // use default DB
|
||||
})
|
||||
} else {
|
||||
fifo.client = redis.NewClusterClient(&redis.ClusterOptions{
|
||||
Addrs: dbs,
|
||||
})
|
||||
}
|
||||
fifo.client = nosql.GetManager().GetRedisClient(config.ConnectionString)
|
||||
if err := fifo.client.Ping().Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@ -5,6 +5,8 @@
|
||||
package queue
|
||||
|
||||
import (
|
||||
"code.gitea.io/gitea/modules/nosql"
|
||||
|
||||
"gitea.com/lunny/levelqueue"
|
||||
)
|
||||
|
||||
@ -15,6 +17,8 @@ const LevelUniqueQueueType Type = "unique-level"
|
||||
type LevelUniqueQueueConfiguration struct {
|
||||
ByteFIFOQueueConfiguration
|
||||
DataDir string
|
||||
ConnectionString string
|
||||
QueueName string
|
||||
}
|
||||
|
||||
// LevelUniqueQueue implements a disk library queue
|
||||
@ -34,7 +38,11 @@ func NewLevelUniqueQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue,
|
||||
}
|
||||
config := configInterface.(LevelUniqueQueueConfiguration)
|
||||
|
||||
byteFIFO, err := NewLevelUniqueQueueByteFIFO(config.DataDir)
|
||||
if len(config.ConnectionString) == 0 {
|
||||
config.ConnectionString = config.DataDir
|
||||
}
|
||||
|
||||
byteFIFO, err := NewLevelUniqueQueueByteFIFO(config.ConnectionString, config.QueueName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@ -56,16 +64,23 @@ var _ (UniqueByteFIFO) = &LevelUniqueQueueByteFIFO{}
|
||||
// LevelUniqueQueueByteFIFO represents a ByteFIFO formed from a LevelUniqueQueue
|
||||
type LevelUniqueQueueByteFIFO struct {
|
||||
internal *levelqueue.UniqueQueue
|
||||
connection string
|
||||
}
|
||||
|
||||
// NewLevelUniqueQueueByteFIFO creates a new ByteFIFO formed from a LevelUniqueQueue
|
||||
func NewLevelUniqueQueueByteFIFO(dataDir string) (*LevelUniqueQueueByteFIFO, error) {
|
||||
internal, err := levelqueue.OpenUnique(dataDir)
|
||||
func NewLevelUniqueQueueByteFIFO(connection, prefix string) (*LevelUniqueQueueByteFIFO, error) {
|
||||
db, err := nosql.GetManager().GetLevelDB(connection)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
internal, err := levelqueue.NewUniqueQueue(db, []byte(prefix), []byte(prefix+"-unique"), false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &LevelUniqueQueueByteFIFO{
|
||||
connection: connection,
|
||||
internal: internal,
|
||||
}, nil
|
||||
}
|
||||
@ -96,7 +111,9 @@ func (fifo *LevelUniqueQueueByteFIFO) Has(data []byte) (bool, error) {
|
||||
|
||||
// Close this fifo
|
||||
func (fifo *LevelUniqueQueueByteFIFO) Close() error {
|
||||
return fifo.internal.Close()
|
||||
err := fifo.internal.Close()
|
||||
_ = nosql.GetManager().CloseLevelDB(fifo.connection)
|
||||
return err
|
||||
}
|
||||
|
||||
func init() {
|
||||
|
||||
@ -4,7 +4,7 @@
|
||||
|
||||
package queue
|
||||
|
||||
import "github.com/go-redis/redis"
|
||||
import "github.com/go-redis/redis/v7"
|
||||
|
||||
// RedisUniqueQueueType is the type for redis queue
|
||||
const RedisUniqueQueueType Type = "unique-redis"
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
// Copyright 2013 Beego Authors
|
||||
// Copyright 2014 The Macaron Authors
|
||||
// Copyright 2020 The Gitea Authors. All rights reserved.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License"): you may
|
||||
// not use this file except in compliance with the License. You may obtain
|
||||
@ -17,19 +18,18 @@ package session
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"code.gitea.io/gitea/modules/nosql"
|
||||
|
||||
"gitea.com/macaron/session"
|
||||
"github.com/go-redis/redis"
|
||||
"github.com/unknwon/com"
|
||||
"gopkg.in/ini.v1"
|
||||
"github.com/go-redis/redis/v7"
|
||||
)
|
||||
|
||||
// RedisStore represents a redis session store implementation.
|
||||
type RedisStore struct {
|
||||
c *redis.Client
|
||||
c redis.UniversalClient
|
||||
prefix, sid string
|
||||
duration time.Duration
|
||||
lock sync.RWMutex
|
||||
@ -37,7 +37,7 @@ type RedisStore struct {
|
||||
}
|
||||
|
||||
// NewRedisStore creates and returns a redis session store.
|
||||
func NewRedisStore(c *redis.Client, prefix, sid string, dur time.Duration, kv map[interface{}]interface{}) *RedisStore {
|
||||
func NewRedisStore(c redis.UniversalClient, prefix, sid string, dur time.Duration, kv map[interface{}]interface{}) *RedisStore {
|
||||
return &RedisStore{
|
||||
c: c,
|
||||
prefix: prefix,
|
||||
@ -104,7 +104,7 @@ func (s *RedisStore) Flush() error {
|
||||
|
||||
// RedisProvider represents a redis session provider implementation.
|
||||
type RedisProvider struct {
|
||||
c *redis.Client
|
||||
c redis.UniversalClient
|
||||
duration time.Duration
|
||||
prefix string
|
||||
}
|
||||
@ -117,39 +117,16 @@ func (p *RedisProvider) Init(maxlifetime int64, configs string) (err error) {
|
||||
return err
|
||||
}
|
||||
|
||||
cfg, err := ini.Load([]byte(strings.Replace(configs, ",", "\n", -1)))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
uri := nosql.ToRedisURI(configs)
|
||||
|
||||
opt := &redis.Options{
|
||||
Network: "tcp",
|
||||
}
|
||||
for k, v := range cfg.Section("").KeysHash() {
|
||||
for k, v := range uri.Query() {
|
||||
switch k {
|
||||
case "network":
|
||||
opt.Network = v
|
||||
case "addr":
|
||||
opt.Addr = v
|
||||
case "password":
|
||||
opt.Password = v
|
||||
case "db":
|
||||
opt.DB = com.StrTo(v).MustInt()
|
||||
case "pool_size":
|
||||
opt.PoolSize = com.StrTo(v).MustInt()
|
||||
case "idle_timeout":
|
||||
opt.IdleTimeout, err = time.ParseDuration(v + "s")
|
||||
if err != nil {
|
||||
return fmt.Errorf("error parsing idle timeout: %v", err)
|
||||
}
|
||||
case "prefix":
|
||||
p.prefix = v
|
||||
default:
|
||||
return fmt.Errorf("session/redis: unsupported option '%s'", k)
|
||||
p.prefix = v[0]
|
||||
}
|
||||
}
|
||||
|
||||
p.c = redis.NewClient(opt)
|
||||
p.c = nosql.GetManager().GetRedisClient(uri.String())
|
||||
return p.c.Ping().Err()
|
||||
}
|
||||
|
||||
@ -228,11 +205,11 @@ func (p *RedisProvider) Regenerate(oldsid, sid string) (_ session.RawStore, err
|
||||
|
||||
// Count counts and returns number of sessions.
|
||||
func (p *RedisProvider) Count() int {
|
||||
return int(p.c.DbSize().Val())
|
||||
return int(p.c.DBSize().Val())
|
||||
}
|
||||
|
||||
// GC calls GC to clean expired sessions.
|
||||
func (_ *RedisProvider) GC() {}
|
||||
func (*RedisProvider) GC() {}
|
||||
|
||||
func init() {
|
||||
session.Register("redis", &RedisProvider{})
|
||||
@ -15,7 +15,6 @@ import (
|
||||
mysql "gitea.com/macaron/session/mysql"
|
||||
nodb "gitea.com/macaron/session/nodb"
|
||||
postgres "gitea.com/macaron/session/postgres"
|
||||
redis "gitea.com/macaron/session/redis"
|
||||
)
|
||||
|
||||
// VirtualSessionProvider represents a shadowed session provider implementation.
|
||||
@ -40,7 +39,7 @@ func (o *VirtualSessionProvider) Init(gclifetime int64, config string) error {
|
||||
case "file":
|
||||
o.provider = &session.FileProvider{}
|
||||
case "redis":
|
||||
o.provider = &redis.RedisProvider{}
|
||||
o.provider = &RedisProvider{}
|
||||
case "mysql":
|
||||
o.provider = &mysql.MysqlProvider{}
|
||||
case "postgres":
|
||||
|
||||
1
vendor/gitea.com/macaron/cache/redis/redis.goconvey
generated
vendored
1
vendor/gitea.com/macaron/cache/redis/redis.goconvey
generated
vendored
@ -1 +0,0 @@
|
||||
ignore
|
||||
1
vendor/gitea.com/macaron/session/redis/redis.goconvey
generated
vendored
1
vendor/gitea.com/macaron/session/redis/redis.goconvey
generated
vendored
@ -1 +0,0 @@
|
||||
ignore
|
||||
19
vendor/github.com/go-redis/redis/.travis.yml
generated
vendored
19
vendor/github.com/go-redis/redis/.travis.yml
generated
vendored
@ -1,19 +0,0 @@
|
||||
sudo: false
|
||||
language: go
|
||||
|
||||
services:
|
||||
- redis-server
|
||||
|
||||
go:
|
||||
- 1.9.x
|
||||
- 1.10.x
|
||||
- 1.11.x
|
||||
- tip
|
||||
|
||||
matrix:
|
||||
allow_failures:
|
||||
- go: tip
|
||||
|
||||
install:
|
||||
- go get github.com/onsi/ginkgo
|
||||
- go get github.com/onsi/gomega
|
||||
25
vendor/github.com/go-redis/redis/CHANGELOG.md
generated
vendored
25
vendor/github.com/go-redis/redis/CHANGELOG.md
generated
vendored
@ -1,25 +0,0 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Cluster and Ring pipelines process commands for each node in its own goroutine.
|
||||
|
||||
## 6.14
|
||||
|
||||
- Added Options.MinIdleConns.
|
||||
- Added Options.MaxConnAge.
|
||||
- PoolStats.FreeConns is renamed to PoolStats.IdleConns.
|
||||
- Add Client.Do to simplify creating custom commands.
|
||||
- Add Cmd.String, Cmd.Int, Cmd.Int64, Cmd.Uint64, Cmd.Float64, and Cmd.Bool helpers.
|
||||
- Lower memory usage.
|
||||
|
||||
## v6.13
|
||||
|
||||
- Ring got new options called `HashReplicas` and `Hash`. It is recommended to set `HashReplicas = 1000` for better keys distribution between shards.
|
||||
- Cluster client was optimized to use much less memory when reloading cluster state.
|
||||
- PubSub.ReceiveMessage is re-worked to not use ReceiveTimeout so it does not lose data when timeout occurres. In most cases it is recommended to use PubSub.Channel instead.
|
||||
- Dialer.KeepAlive is set to 5 minutes by default.
|
||||
|
||||
## v6.12
|
||||
|
||||
- ClusterClient got new option called `ClusterSlots` which allows to build cluster of normal Redis Servers that don't have cluster mode enabled. See https://godoc.org/github.com/go-redis/redis#example-NewClusterClient--ManualSetup
|
||||
89
vendor/github.com/go-redis/redis/internal/error.go
generated
vendored
89
vendor/github.com/go-redis/redis/internal/error.go
generated
vendored
@ -1,89 +0,0 @@
|
||||
package internal
|
||||
|
||||
import (
|
||||
"io"
|
||||
"net"
|
||||
"strings"
|
||||
|
||||
"github.com/go-redis/redis/internal/proto"
|
||||
)
|
||||
|
||||
func IsRetryableError(err error, retryTimeout bool) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
if err == io.EOF {
|
||||
return true
|
||||
}
|
||||
if netErr, ok := err.(net.Error); ok {
|
||||
if netErr.Timeout() {
|
||||
return retryTimeout
|
||||
}
|
||||
return true
|
||||
}
|
||||
s := err.Error()
|
||||
if s == "ERR max number of clients reached" {
|
||||
return true
|
||||
}
|
||||
if strings.HasPrefix(s, "LOADING ") {
|
||||
return true
|
||||
}
|
||||
if strings.HasPrefix(s, "READONLY ") {
|
||||
return true
|
||||
}
|
||||
if strings.HasPrefix(s, "CLUSTERDOWN ") {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func IsRedisError(err error) bool {
|
||||
_, ok := err.(proto.RedisError)
|
||||
return ok
|
||||
}
|
||||
|
||||
func IsBadConn(err error, allowTimeout bool) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
if IsRedisError(err) {
|
||||
// #790
|
||||
return IsReadOnlyError(err)
|
||||
}
|
||||
if allowTimeout {
|
||||
if netErr, ok := err.(net.Error); ok && netErr.Timeout() {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func IsMovedError(err error) (moved bool, ask bool, addr string) {
|
||||
if !IsRedisError(err) {
|
||||
return
|
||||
}
|
||||
|
||||
s := err.Error()
|
||||
if strings.HasPrefix(s, "MOVED ") {
|
||||
moved = true
|
||||
} else if strings.HasPrefix(s, "ASK ") {
|
||||
ask = true
|
||||
} else {
|
||||
return
|
||||
}
|
||||
|
||||
ind := strings.LastIndex(s, " ")
|
||||
if ind == -1 {
|
||||
return false, false, ""
|
||||
}
|
||||
addr = s[ind+1:]
|
||||
return
|
||||
}
|
||||
|
||||
func IsLoadingError(err error) bool {
|
||||
return strings.HasPrefix(err.Error(), "LOADING ")
|
||||
}
|
||||
|
||||
func IsReadOnlyError(err error) bool {
|
||||
return strings.HasPrefix(err.Error(), "READONLY ")
|
||||
}
|
||||
15
vendor/github.com/go-redis/redis/internal/log.go
generated
vendored
15
vendor/github.com/go-redis/redis/internal/log.go
generated
vendored
@ -1,15 +0,0 @@
|
||||
package internal
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
)
|
||||
|
||||
var Logger *log.Logger
|
||||
|
||||
func Logf(s string, args ...interface{}) {
|
||||
if Logger == nil {
|
||||
return
|
||||
}
|
||||
Logger.Output(2, fmt.Sprintf(s, args...))
|
||||
}
|
||||
93
vendor/github.com/go-redis/redis/internal/pool/conn.go
generated
vendored
93
vendor/github.com/go-redis/redis/internal/pool/conn.go
generated
vendored
@ -1,93 +0,0 @@
|
||||
package pool
|
||||
|
||||
import (
|
||||
"net"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/go-redis/redis/internal/proto"
|
||||
)
|
||||
|
||||
var noDeadline = time.Time{}
|
||||
|
||||
type Conn struct {
|
||||
netConn net.Conn
|
||||
|
||||
rd *proto.Reader
|
||||
rdLocked bool
|
||||
wr *proto.Writer
|
||||
|
||||
InitedAt time.Time
|
||||
pooled bool
|
||||
usedAt atomic.Value
|
||||
}
|
||||
|
||||
func NewConn(netConn net.Conn) *Conn {
|
||||
cn := &Conn{
|
||||
netConn: netConn,
|
||||
}
|
||||
cn.rd = proto.NewReader(netConn)
|
||||
cn.wr = proto.NewWriter(netConn)
|
||||
cn.SetUsedAt(time.Now())
|
||||
return cn
|
||||
}
|
||||
|
||||
func (cn *Conn) UsedAt() time.Time {
|
||||
return cn.usedAt.Load().(time.Time)
|
||||
}
|
||||
|
||||
func (cn *Conn) SetUsedAt(tm time.Time) {
|
||||
cn.usedAt.Store(tm)
|
||||
}
|
||||
|
||||
func (cn *Conn) SetNetConn(netConn net.Conn) {
|
||||
cn.netConn = netConn
|
||||
cn.rd.Reset(netConn)
|
||||
cn.wr.Reset(netConn)
|
||||
}
|
||||
|
||||
func (cn *Conn) setReadTimeout(timeout time.Duration) error {
|
||||
now := time.Now()
|
||||
cn.SetUsedAt(now)
|
||||
if timeout > 0 {
|
||||
return cn.netConn.SetReadDeadline(now.Add(timeout))
|
||||
}
|
||||
return cn.netConn.SetReadDeadline(noDeadline)
|
||||
}
|
||||
|
||||
func (cn *Conn) setWriteTimeout(timeout time.Duration) error {
|
||||
now := time.Now()
|
||||
cn.SetUsedAt(now)
|
||||
if timeout > 0 {
|
||||
return cn.netConn.SetWriteDeadline(now.Add(timeout))
|
||||
}
|
||||
return cn.netConn.SetWriteDeadline(noDeadline)
|
||||
}
|
||||
|
||||
func (cn *Conn) Write(b []byte) (int, error) {
|
||||
return cn.netConn.Write(b)
|
||||
}
|
||||
|
||||
func (cn *Conn) RemoteAddr() net.Addr {
|
||||
return cn.netConn.RemoteAddr()
|
||||
}
|
||||
|
||||
func (cn *Conn) WithReader(timeout time.Duration, fn func(rd *proto.Reader) error) error {
|
||||
_ = cn.setReadTimeout(timeout)
|
||||
return fn(cn.rd)
|
||||
}
|
||||
|
||||
func (cn *Conn) WithWriter(timeout time.Duration, fn func(wr *proto.Writer) error) error {
|
||||
_ = cn.setWriteTimeout(timeout)
|
||||
|
||||
firstErr := fn(cn.wr)
|
||||
err := cn.wr.Flush()
|
||||
if err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
|
||||
func (cn *Conn) Close() error {
|
||||
return cn.netConn.Close()
|
||||
}
|
||||
53
vendor/github.com/go-redis/redis/internal/pool/pool_single.go
generated
vendored
53
vendor/github.com/go-redis/redis/internal/pool/pool_single.go
generated
vendored
@ -1,53 +0,0 @@
|
||||
package pool
|
||||
|
||||
type SingleConnPool struct {
|
||||
cn *Conn
|
||||
}
|
||||
|
||||
var _ Pooler = (*SingleConnPool)(nil)
|
||||
|
||||
func NewSingleConnPool(cn *Conn) *SingleConnPool {
|
||||
return &SingleConnPool{
|
||||
cn: cn,
|
||||
}
|
||||
}
|
||||
|
||||
func (p *SingleConnPool) NewConn() (*Conn, error) {
|
||||
panic("not implemented")
|
||||
}
|
||||
|
||||
func (p *SingleConnPool) CloseConn(*Conn) error {
|
||||
panic("not implemented")
|
||||
}
|
||||
|
||||
func (p *SingleConnPool) Get() (*Conn, error) {
|
||||
return p.cn, nil
|
||||
}
|
||||
|
||||
func (p *SingleConnPool) Put(cn *Conn) {
|
||||
if p.cn != cn {
|
||||
panic("p.cn != cn")
|
||||
}
|
||||
}
|
||||
|
||||
func (p *SingleConnPool) Remove(cn *Conn) {
|
||||
if p.cn != cn {
|
||||
panic("p.cn != cn")
|
||||
}
|
||||
}
|
||||
|
||||
func (p *SingleConnPool) Len() int {
|
||||
return 1
|
||||
}
|
||||
|
||||
func (p *SingleConnPool) IdleLen() int {
|
||||
return 0
|
||||
}
|
||||
|
||||
func (p *SingleConnPool) Stats() *Stats {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *SingleConnPool) Close() error {
|
||||
return nil
|
||||
}
|
||||
29
vendor/github.com/go-redis/redis/internal/util.go
generated
vendored
29
vendor/github.com/go-redis/redis/internal/util.go
generated
vendored
@ -1,29 +0,0 @@
|
||||
package internal
|
||||
|
||||
import "github.com/go-redis/redis/internal/util"
|
||||
|
||||
func ToLower(s string) string {
|
||||
if isLower(s) {
|
||||
return s
|
||||
}
|
||||
|
||||
b := make([]byte, len(s))
|
||||
for i := range b {
|
||||
c := s[i]
|
||||
if c >= 'A' && c <= 'Z' {
|
||||
c += 'a' - 'A'
|
||||
}
|
||||
b[i] = c
|
||||
}
|
||||
return util.BytesToString(b)
|
||||
}
|
||||
|
||||
func isLower(s string) bool {
|
||||
for i := 0; i < len(s); i++ {
|
||||
c := s[i]
|
||||
if c >= 'A' && c <= 'Z' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
580
vendor/github.com/go-redis/redis/redis.go
generated
vendored
580
vendor/github.com/go-redis/redis/redis.go
generated
vendored
File diff suppressed because it is too large
Load Diff
15
vendor/github.com/go-redis/redis/v7/.golangci.yml
generated
vendored
Normal file
15
vendor/github.com/go-redis/redis/v7/.golangci.yml
generated
vendored
Normal file
@ -0,0 +1,15 @@
|
||||
run:
|
||||
concurrency: 8
|
||||
deadline: 5m
|
||||
tests: false
|
||||
linters:
|
||||
enable-all: true
|
||||
disable:
|
||||
- funlen
|
||||
- gochecknoglobals
|
||||
- gocognit
|
||||
- goconst
|
||||
- godox
|
||||
- gosec
|
||||
- maligned
|
||||
- wsl
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user