Migrate to dep (#3972)
* Update makefile to use dep * Migrate to dep * Fix some deps * Try to find a better version for golang.org/x/net * Try to find a better version for golang.org/x/oauth2
This commit is contained in:
Antoine GIRARD
committed by
Lauris BH
Lauris BH
parent
d7fd9bf7bb
commit
3f3383dc0a
126
vendor/github.com/PuerkitoBio/goquery/README.md
generated
vendored
126
vendor/github.com/PuerkitoBio/goquery/README.md
generated
vendored
@ -1,126 +0,0 @@
|
||||
# goquery - a little like that j-thing, only in Go
|
||||
[](http://travis-ci.org/PuerkitoBio/goquery) [](http://godoc.org/github.com/PuerkitoBio/goquery) [](https://sourcegraph.com/github.com/PuerkitoBio/goquery?badge)
|
||||
|
||||
|
||||
goquery brings a syntax and a set of features similar to [jQuery][] to the [Go language][go]. It is based on Go's [net/html package][html] and the CSS Selector library [cascadia][]. Since the net/html parser returns nodes, and not a full-featured DOM tree, jQuery's stateful manipulation functions (like height(), css(), detach()) have been left off.
|
||||
|
||||
Also, because the net/html parser requires UTF-8 encoding, so does goquery: it is the caller's responsibility to ensure that the source document provides UTF-8 encoded HTML. See the [wiki][] for various options to do this.
|
||||
|
||||
Syntax-wise, it is as close as possible to jQuery, with the same function names when possible, and that warm and fuzzy chainable interface. jQuery being the ultra-popular library that it is, I felt that writing a similar HTML-manipulating library was better to follow its API than to start anew (in the same spirit as Go's `fmt` package), even though some of its methods are less than intuitive (looking at you, [index()][index]...).
|
||||
|
||||
## Installation
|
||||
|
||||
Please note that because of the net/html dependency, goquery requires Go1.1+.
|
||||
|
||||
$ go get github.com/PuerkitoBio/goquery
|
||||
|
||||
(optional) To run unit tests:
|
||||
|
||||
$ cd $GOPATH/src/github.com/PuerkitoBio/goquery
|
||||
$ go test
|
||||
|
||||
(optional) To run benchmarks (warning: it runs for a few minutes):
|
||||
|
||||
$ cd $GOPATH/src/github.com/PuerkitoBio/goquery
|
||||
$ go test -bench=".*"
|
||||
|
||||
## Changelog
|
||||
|
||||
**Note that goquery's API is now stable, and will not break.**
|
||||
|
||||
* **2017-02-12 (v1.1.0)** : Add `SetHtml` and `SetText` (thanks to @glebtv).
|
||||
* **2016-12-29 (v1.0.2)** : Optimize allocations for `Selection.Text` (thanks to @radovskyb).
|
||||
* **2016-08-28 (v1.0.1)** : Optimize performance for large documents.
|
||||
* **2016-07-27 (v1.0.0)** : Tag version 1.0.0.
|
||||
* **2016-06-15** : Invalid selector strings internally compile to a `Matcher` implementation that never matches any node (instead of a panic). So for example, `doc.Find("~")` returns an empty `*Selection` object.
|
||||
* **2016-02-02** : Add `NodeName` utility function similar to the DOM's `nodeName` property. It returns the tag name of the first element in a selection, and other relevant values of non-element nodes (see godoc for details). Add `OuterHtml` utility function similar to the DOM's `outerHTML` property (named `OuterHtml` in small caps for consistency with the existing `Html` method on the `Selection`).
|
||||
* **2015-04-20** : Add `AttrOr` helper method to return the attribute's value or a default value if absent. Thanks to [piotrkowalczuk][piotr].
|
||||
* **2015-02-04** : Add more manipulation functions - Prepend* - thanks again to [Andrew Stone][thatguystone].
|
||||
* **2014-11-28** : Add more manipulation functions - ReplaceWith*, Wrap* and Unwrap - thanks again to [Andrew Stone][thatguystone].
|
||||
* **2014-11-07** : Add manipulation functions (thanks to [Andrew Stone][thatguystone]) and `*Matcher` functions, that receive compiled cascadia selectors instead of selector strings, thus avoiding potential panics thrown by goquery via `cascadia.MustCompile` calls. This results in better performance (selectors can be compiled once and reused) and more idiomatic error handling (you can handle cascadia's compilation errors, instead of recovering from panics, which had been bugging me for a long time). Note that the actual type expected is a `Matcher` interface, that `cascadia.Selector` implements. Other matcher implementations could be used.
|
||||
* **2014-11-06** : Change import paths of net/html to golang.org/x/net/html (see https://groups.google.com/forum/#!topic/golang-nuts/eD8dh3T9yyA). Make sure to update your code to use the new import path too when you call goquery with `html.Node`s.
|
||||
* **v0.3.2** : Add `NewDocumentFromReader()` (thanks jweir) which allows creating a goquery document from an io.Reader.
|
||||
* **v0.3.1** : Add `NewDocumentFromResponse()` (thanks assassingj) which allows creating a goquery document from an http response.
|
||||
* **v0.3.0** : Add `EachWithBreak()` which allows to break out of an `Each()` loop by returning false. This function was added instead of changing the existing `Each()` to avoid breaking compatibility.
|
||||
* **v0.2.1** : Make go-getable, now that [go.net/html is Go1.0-compatible][gonet] (thanks to @matrixik for pointing this out).
|
||||
* **v0.2.0** : Add support for negative indices in Slice(). **BREAKING CHANGE** `Document.Root` is removed, `Document` is now a `Selection` itself (a selection of one, the root element, just like `Document.Root` was before). Add jQuery's Closest() method.
|
||||
* **v0.1.1** : Add benchmarks to use as baseline for refactorings, refactor Next...() and Prev...() methods to use the new html package's linked list features (Next/PrevSibling, FirstChild). Good performance boost (40+% in some cases).
|
||||
* **v0.1.0** : Initial release.
|
||||
|
||||
## API
|
||||
|
||||
goquery exposes two structs, `Document` and `Selection`, and the `Matcher` interface. Unlike jQuery, which is loaded as part of a DOM document, and thus acts on its containing document, goquery doesn't know which HTML document to act upon. So it needs to be told, and that's what the `Document` type is for. It holds the root document node as the initial Selection value to manipulate.
|
||||
|
||||
jQuery often has many variants for the same function (no argument, a selector string argument, a jQuery object argument, a DOM element argument, ...). Instead of exposing the same features in goquery as a single method with variadic empty interface arguments, statically-typed signatures are used following this naming convention:
|
||||
|
||||
* When the jQuery equivalent can be called with no argument, it has the same name as jQuery for the no argument signature (e.g.: `Prev()`), and the version with a selector string argument is called `XxxFiltered()` (e.g.: `PrevFiltered()`)
|
||||
* When the jQuery equivalent **requires** one argument, the same name as jQuery is used for the selector string version (e.g.: `Is()`)
|
||||
* The signatures accepting a jQuery object as argument are defined in goquery as `XxxSelection()` and take a `*Selection` object as argument (e.g.: `FilterSelection()`)
|
||||
* The signatures accepting a DOM element as argument in jQuery are defined in goquery as `XxxNodes()` and take a variadic argument of type `*html.Node` (e.g.: `FilterNodes()`)
|
||||
* The signatures accepting a function as argument in jQuery are defined in goquery as `XxxFunction()` and take a function as argument (e.g.: `FilterFunction()`)
|
||||
* The goquery methods that can be called with a selector string have a corresponding version that take a `Matcher` interface and are defined as `XxxMatcher()` (e.g.: `IsMatcher()`)
|
||||
|
||||
Utility functions that are not in jQuery but are useful in Go are implemented as functions (that take a `*Selection` as parameter), to avoid a potential naming clash on the `*Selection`'s methods (reserved for jQuery-equivalent behaviour).
|
||||
|
||||
The complete [godoc reference documentation can be found here][doc].
|
||||
|
||||
Please note that Cascadia's selectors do not necessarily match all supported selectors of jQuery (Sizzle). See the [cascadia project][cascadia] for details. Invalid selector strings compile to a `Matcher` that fails to match any node. Behaviour of the various functions that take a selector string as argument follows from that fact, e.g. (where `~` is an invalid selector string):
|
||||
|
||||
* `Find("~")` returns an empty selection because the selector string doesn't match anything.
|
||||
* `Add("~")` returns a new selection that holds the same nodes as the original selection, because it didn't add any node (selector string didn't match anything).
|
||||
* `ParentsFiltered("~")` returns an empty selection because the selector string doesn't match anything.
|
||||
* `ParentsUntil("~")` returns all parents of the selection because the selector string didn't match any element to stop before the top element.
|
||||
|
||||
## Examples
|
||||
|
||||
See some tips and tricks in the [wiki][].
|
||||
|
||||
Adapted from example_test.go:
|
||||
|
||||
```Go
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"github.com/PuerkitoBio/goquery"
|
||||
)
|
||||
|
||||
func ExampleScrape() {
|
||||
doc, err := goquery.NewDocument("http://metalsucks.net")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Find the review items
|
||||
doc.Find(".sidebar-reviews article .content-block").Each(func(i int, s *goquery.Selection) {
|
||||
// For each item found, get the band and title
|
||||
band := s.Find("a").Text()
|
||||
title := s.Find("i").Text()
|
||||
fmt.Printf("Review %d: %s - %s\n", i, band, title)
|
||||
})
|
||||
}
|
||||
|
||||
func main() {
|
||||
ExampleScrape()
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
The [BSD 3-Clause license][bsd], the same as the [Go language][golic]. Cascadia's license is [here][caslic].
|
||||
|
||||
[jquery]: http://jquery.com/
|
||||
[go]: http://golang.org/
|
||||
[cascadia]: https://github.com/andybalholm/cascadia
|
||||
[bsd]: http://opensource.org/licenses/BSD-3-Clause
|
||||
[golic]: http://golang.org/LICENSE
|
||||
[caslic]: https://github.com/andybalholm/cascadia/blob/master/LICENSE
|
||||
[doc]: http://godoc.org/github.com/PuerkitoBio/goquery
|
||||
[index]: http://api.jquery.com/index/
|
||||
[gonet]: https://github.com/golang/net/
|
||||
[html]: http://godoc.org/golang.org/x/net/html
|
||||
[wiki]: https://github.com/PuerkitoBio/goquery/wiki/Tips-and-tricks
|
||||
[thatguystone]: https://github.com/thatguystone
|
||||
[piotr]: https://github.com/piotrkowalczuk
|
||||
121
vendor/github.com/RoaringBitmap/roaring/Makefile
generated
vendored
121
vendor/github.com/RoaringBitmap/roaring/Makefile
generated
vendored
@ -1,121 +0,0 @@
|
||||
.PHONY: help all test format fmtcheck vet lint qa deps clean nuke rle backrle ser fetch-real-roaring-datasets
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
# Display general help about this command
|
||||
help:
|
||||
@echo ""
|
||||
@echo "The following commands are available:"
|
||||
@echo ""
|
||||
@echo " make qa : Run all the tests"
|
||||
@echo " make test : Run the unit tests"
|
||||
@echo ""
|
||||
@echo " make format : Format the source code"
|
||||
@echo " make fmtcheck : Check if the source code has been formatted"
|
||||
@echo " make vet : Check for suspicious constructs"
|
||||
@echo " make lint : Check for style errors"
|
||||
@echo ""
|
||||
@echo " make deps : Get the dependencies"
|
||||
@echo " make clean : Remove any build artifact"
|
||||
@echo " make nuke : Deletes any intermediate file"
|
||||
@echo ""
|
||||
@echo " make fuzz-smat : Fuzzy testing with smat"
|
||||
@echo " make fuzz-stream : Fuzzy testing with stream deserialization"
|
||||
@echo " make fuzz-buffer : Fuzzy testing with buffer deserialization"
|
||||
@echo ""
|
||||
|
||||
# Alias for help target
|
||||
all: help
|
||||
test:
|
||||
go test
|
||||
go test -race -run TestConcurrent*
|
||||
# Format the source code
|
||||
format:
|
||||
@find ./ -type f -name "*.go" -exec gofmt -w {} \;
|
||||
|
||||
# Check if the source code has been formatted
|
||||
fmtcheck:
|
||||
@mkdir -p target
|
||||
@find ./ -type f -name "*.go" -exec gofmt -d {} \; | tee target/format.diff
|
||||
@test ! -s target/format.diff || { echo "ERROR: the source code has not been formatted - please use 'make format' or 'gofmt'"; exit 1; }
|
||||
|
||||
# Check for syntax errors
|
||||
vet:
|
||||
GOPATH=$(GOPATH) go vet ./...
|
||||
|
||||
# Check for style errors
|
||||
lint:
|
||||
GOPATH=$(GOPATH) PATH=$(GOPATH)/bin:$(PATH) golint ./...
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
# Alias to run all quality-assurance checks
|
||||
qa: fmtcheck test vet lint
|
||||
|
||||
# --- INSTALL ---
|
||||
|
||||
# Get the dependencies
|
||||
deps:
|
||||
GOPATH=$(GOPATH) go get github.com/smartystreets/goconvey/convey
|
||||
GOPATH=$(GOPATH) go get github.com/willf/bitset
|
||||
GOPATH=$(GOPATH) go get github.com/golang/lint/golint
|
||||
GOPATH=$(GOPATH) go get github.com/mschoch/smat
|
||||
GOPATH=$(GOPATH) go get github.com/dvyukov/go-fuzz/go-fuzz
|
||||
GOPATH=$(GOPATH) go get github.com/dvyukov/go-fuzz/go-fuzz-build
|
||||
GOPATH=$(GOPATH) go get github.com/glycerine/go-unsnap-stream
|
||||
GOPATH=$(GOPATH) go get github.com/philhofer/fwd
|
||||
GOPATH=$(GOPATH) go get github.com/jtolds/gls
|
||||
|
||||
fuzz-smat:
|
||||
go test -tags=gofuzz -run=TestGenerateSmatCorpus
|
||||
go-fuzz-build -func FuzzSmat github.com/RoaringBitmap/roaring
|
||||
go-fuzz -bin=./roaring-fuzz.zip -workdir=workdir/ -timeout=200
|
||||
|
||||
|
||||
fuzz-stream:
|
||||
go-fuzz-build -func FuzzSerializationStream github.com/RoaringBitmap/roaring
|
||||
go-fuzz -bin=./roaring-fuzz.zip -workdir=workdir/ -timeout=200
|
||||
|
||||
|
||||
fuzz-buffer:
|
||||
go-fuzz-build -func FuzzSerializationBuffer github.com/RoaringBitmap/roaring
|
||||
go-fuzz -bin=./roaring-fuzz.zip -workdir=workdir/ -timeout=200
|
||||
|
||||
# Remove any build artifact
|
||||
clean:
|
||||
GOPATH=$(GOPATH) go clean ./...
|
||||
|
||||
# Deletes any intermediate file
|
||||
nuke:
|
||||
rm -rf ./target
|
||||
GOPATH=$(GOPATH) go clean -i ./...
|
||||
|
||||
rle:
|
||||
cp rle.go rle16.go
|
||||
perl -pi -e 's/32/16/g' rle16.go
|
||||
cp rle_test.go rle16_test.go
|
||||
perl -pi -e 's/32/16/g' rle16_test.go
|
||||
|
||||
backrle:
|
||||
cp rle16.go rle.go
|
||||
perl -pi -e 's/16/32/g' rle.go
|
||||
perl -pi -e 's/2032/2016/g' rle.go
|
||||
|
||||
ser: rle
|
||||
go generate
|
||||
|
||||
cover:
|
||||
go test -coverprofile=coverage.out
|
||||
go tool cover -html=coverage.out
|
||||
|
||||
fetch-real-roaring-datasets:
|
||||
# pull github.com/RoaringBitmap/real-roaring-datasets -> testdata/real-roaring-datasets
|
||||
git submodule init
|
||||
git submodule update
|
||||
246
vendor/github.com/RoaringBitmap/roaring/README.md
generated
vendored
246
vendor/github.com/RoaringBitmap/roaring/README.md
generated
vendored
@ -1,246 +0,0 @@
|
||||
roaring [](https://travis-ci.org/RoaringBitmap/roaring) [](https://coveralls.io/github/RoaringBitmap/roaring?branch=master) [](https://godoc.org/github.com/RoaringBitmap/roaring) [](https://goreportcard.com/report/github.com/RoaringBitmap/roaring)
|
||||
=============
|
||||
|
||||
This is a go version of the Roaring bitmap data structure.
|
||||
|
||||
|
||||
|
||||
Roaring bitmaps are used by several major systems such as [Apache Lucene][lucene] and derivative systems such as [Solr][solr] and
|
||||
[Elasticsearch][elasticsearch], [Metamarkets' Druid][druid], [LinkedIn Pinot][pinot], [Netflix Atlas][atlas], [Apache Spark][spark], [OpenSearchServer][opensearchserver], [Cloud Torrent][cloudtorrent], [Whoosh][whoosh], [Pilosa][pilosa], [Microsoft Visual Studio Team Services (VSTS)][vsts], and eBay's [Apache Kylin][kylin].
|
||||
|
||||
[lucene]: https://lucene.apache.org/
|
||||
[solr]: https://lucene.apache.org/solr/
|
||||
[elasticsearch]: https://www.elastic.co/products/elasticsearch
|
||||
[druid]: http://druid.io/
|
||||
[spark]: https://spark.apache.org/
|
||||
[opensearchserver]: http://www.opensearchserver.com
|
||||
[cloudtorrent]: https://github.com/jpillora/cloud-torrent
|
||||
[whoosh]: https://bitbucket.org/mchaput/whoosh/wiki/Home
|
||||
[pilosa]: https://www.pilosa.com/
|
||||
[kylin]: http://kylin.apache.org/
|
||||
[pinot]: http://github.com/linkedin/pinot/wiki
|
||||
[vsts]: https://www.visualstudio.com/team-services/
|
||||
[atlas]: https://github.com/Netflix/atlas
|
||||
|
||||
Roaring bitmaps are found to work well in many important applications:
|
||||
|
||||
> Use Roaring for bitmap compression whenever possible. Do not use other bitmap compression methods ([Wang et al., SIGMOD 2017](http://db.ucsd.edu/wp-content/uploads/2017/03/sidm338-wangA.pdf))
|
||||
|
||||
|
||||
The ``roaring`` Go library is used by
|
||||
* [Cloud Torrent](https://github.com/jpillora/cloud-torrent): a self-hosted remote torrent client
|
||||
* [runv](https://github.com/hyperhq/runv): an Hypervisor-based runtime for the Open Containers Initiative
|
||||
* [InfluxDB](https://www.influxdata.com)
|
||||
* [Pilosa](https://www.pilosa.com/)
|
||||
* [Bleve](http://www.blevesearch.com)
|
||||
|
||||
This library is used in production in several systems, it is part of the [Awesome Go collection](https://awesome-go.com).
|
||||
|
||||
|
||||
There are also [Java](https://github.com/RoaringBitmap/RoaringBitmap) and [C/C++](https://github.com/RoaringBitmap/CRoaring) versions. The Java, C, C++ and Go version are binary compatible: e.g, you can save bitmaps
|
||||
from a Java program and load them back in Go, and vice versa. We have a [format specification](https://github.com/RoaringBitmap/RoaringFormatSpec).
|
||||
|
||||
|
||||
This code is licensed under Apache License, Version 2.0 (ASL2.0).
|
||||
|
||||
Copyright 2016-... by the authors.
|
||||
|
||||
|
||||
### References
|
||||
|
||||
- Daniel Lemire, Owen Kaser, Nathan Kurz, Luca Deri, Chris O'Hara, François Saint-Jacques, Gregory Ssi-Yan-Kai, Roaring Bitmaps: Implementation of an Optimized Software Library, Software: Practice and Experience 48 (4), 2018 [arXiv:1709.07821](https://arxiv.org/abs/1709.07821)
|
||||
- Samy Chambi, Daniel Lemire, Owen Kaser, Robert Godin,
|
||||
Better bitmap performance with Roaring bitmaps,
|
||||
Software: Practice and Experience 46 (5), 2016.
|
||||
http://arxiv.org/abs/1402.6407 This paper used data from http://lemire.me/data/realroaring2014.html
|
||||
- Daniel Lemire, Gregory Ssi-Yan-Kai, Owen Kaser, Consistently faster and smaller compressed bitmaps with Roaring, Software: Practice and Experience 46 (11), 2016. http://arxiv.org/abs/1603.06549
|
||||
|
||||
|
||||
### Dependencies
|
||||
|
||||
Dependencies are fetched automatically by giving the `-t` flag to `go get`.
|
||||
|
||||
they include
|
||||
- github.com/smartystreets/goconvey/convey
|
||||
- github.com/willf/bitset
|
||||
- github.com/mschoch/smat
|
||||
- github.com/glycerine/go-unsnap-stream
|
||||
- github.com/philhofer/fwd
|
||||
- github.com/jtolds/gls
|
||||
|
||||
Note that the smat library requires Go 1.6 or better.
|
||||
|
||||
#### Installation
|
||||
|
||||
- go get -t github.com/RoaringBitmap/roaring
|
||||
|
||||
|
||||
### Example
|
||||
|
||||
Here is a simplified but complete example:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/RoaringBitmap/roaring"
|
||||
"bytes"
|
||||
)
|
||||
|
||||
|
||||
func main() {
|
||||
// example inspired by https://github.com/fzandona/goroar
|
||||
fmt.Println("==roaring==")
|
||||
rb1 := roaring.BitmapOf(1, 2, 3, 4, 5, 100, 1000)
|
||||
fmt.Println(rb1.String())
|
||||
|
||||
rb2 := roaring.BitmapOf(3, 4, 1000)
|
||||
fmt.Println(rb2.String())
|
||||
|
||||
rb3 := roaring.New()
|
||||
fmt.Println(rb3.String())
|
||||
|
||||
fmt.Println("Cardinality: ", rb1.GetCardinality())
|
||||
|
||||
fmt.Println("Contains 3? ", rb1.Contains(3))
|
||||
|
||||
rb1.And(rb2)
|
||||
|
||||
rb3.Add(1)
|
||||
rb3.Add(5)
|
||||
|
||||
rb3.Or(rb1)
|
||||
|
||||
// computes union of the three bitmaps in parallel using 4 workers
|
||||
roaring.ParOr(4, rb1, rb2, rb3)
|
||||
// computes intersection of the three bitmaps in parallel using 4 workers
|
||||
roaring.ParAnd(4, rb1, rb2, rb3)
|
||||
|
||||
|
||||
// prints 1, 3, 4, 5, 1000
|
||||
i := rb3.Iterator()
|
||||
for i.HasNext() {
|
||||
fmt.Println(i.Next())
|
||||
}
|
||||
fmt.Println()
|
||||
|
||||
// next we include an example of serialization
|
||||
buf := new(bytes.Buffer)
|
||||
rb1.WriteTo(buf) // we omit error handling
|
||||
newrb:= roaring.New()
|
||||
newrb.ReadFrom(buf)
|
||||
if rb1.Equals(newrb) {
|
||||
fmt.Println("I wrote the content to a byte stream and read it back.")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If you wish to use serialization and handle errors, you might want to
|
||||
consider the following sample of code:
|
||||
|
||||
```go
|
||||
rb := BitmapOf(1, 2, 3, 4, 5, 100, 1000)
|
||||
buf := new(bytes.Buffer)
|
||||
size,err:=rb.WriteTo(buf)
|
||||
if err != nil {
|
||||
t.Errorf("Failed writing")
|
||||
}
|
||||
newrb:= New()
|
||||
size,err=newrb.ReadFrom(buf)
|
||||
if err != nil {
|
||||
t.Errorf("Failed reading")
|
||||
}
|
||||
if ! rb.Equals(newrb) {
|
||||
t.Errorf("Cannot retrieve serialized version")
|
||||
}
|
||||
```
|
||||
|
||||
Given N integers in [0,x), then the serialized size in bytes of
|
||||
a Roaring bitmap should never exceed this bound:
|
||||
|
||||
`` 8 + 9 * ((long)x+65535)/65536 + 2 * N ``
|
||||
|
||||
That is, given a fixed overhead for the universe size (x), Roaring
|
||||
bitmaps never use more than 2 bytes per integer. You can call
|
||||
``BoundSerializedSizeInBytes`` for a more precise estimate.
|
||||
|
||||
|
||||
### Documentation
|
||||
|
||||
Current documentation is available at http://godoc.org/github.com/RoaringBitmap/roaring
|
||||
|
||||
### Goroutine safety
|
||||
|
||||
In general, it should not generally be considered safe to access
|
||||
the same bitmaps using different goroutines--they are left
|
||||
unsynchronized for performance. Should you want to access
|
||||
a Bitmap from more than one goroutine, you should
|
||||
provide synchronization. Typically this is done by using channels to pass
|
||||
the *Bitmap around (in Go style; so there is only ever one owner),
|
||||
or by using `sync.Mutex` to serialize operations on Bitmaps.
|
||||
|
||||
### Coverage
|
||||
|
||||
We test our software. For a report on our test coverage, see
|
||||
|
||||
https://coveralls.io/github/RoaringBitmap/roaring?branch=master
|
||||
|
||||
### Benchmark
|
||||
|
||||
Type
|
||||
|
||||
go test -bench Benchmark -run -
|
||||
|
||||
To run benchmarks on [Real Roaring Datasets](https://github.com/RoaringBitmap/real-roaring-datasets)
|
||||
run the following:
|
||||
|
||||
```sh
|
||||
go get github.com/RoaringBitmap/real-roaring-datasets
|
||||
BENCH_REAL_DATA=1 go test -bench BenchmarkRealData -run -
|
||||
```
|
||||
|
||||
### Iterative use
|
||||
|
||||
You can use roaring with gore:
|
||||
|
||||
- go get -u github.com/motemen/gore
|
||||
- Make sure that ``$GOPATH/bin`` is in your ``$PATH``.
|
||||
- go get github/RoaringBitmap/roaring
|
||||
|
||||
```go
|
||||
$ gore
|
||||
gore version 0.2.6 :help for help
|
||||
gore> :import github.com/RoaringBitmap/roaring
|
||||
gore> x:=roaring.New()
|
||||
gore> x.Add(1)
|
||||
gore> x.String()
|
||||
"{1}"
|
||||
```
|
||||
|
||||
|
||||
### Fuzzy testing
|
||||
|
||||
You can help us test further the library with fuzzy testing:
|
||||
|
||||
go get github.com/dvyukov/go-fuzz/go-fuzz
|
||||
go get github.com/dvyukov/go-fuzz/go-fuzz-build
|
||||
go test -tags=gofuzz -run=TestGenerateSmatCorpus
|
||||
go-fuzz-build github.com/RoaringBitmap/roaring
|
||||
go-fuzz -bin=./roaring-fuzz.zip -workdir=workdir/ -timeout=200
|
||||
|
||||
Let it run, and if the # of crashers is > 0, check out the reports in
|
||||
the workdir where you should be able to find the panic goroutine stack
|
||||
traces.
|
||||
|
||||
### Alternative in Go
|
||||
|
||||
There is a Go version wrapping the C/C++ implementation https://github.com/RoaringBitmap/gocroaring
|
||||
|
||||
For an alternative implementation in Go, see https://github.com/fzandona/goroar
|
||||
The two versions were written independently.
|
||||
|
||||
|
||||
### Mailing list/discussion group
|
||||
|
||||
https://groups.google.com/forum/#!forum/roaring-bitmaps
|
||||
67
vendor/github.com/Smerity/govarint/README.md
generated
vendored
67
vendor/github.com/Smerity/govarint/README.md
generated
vendored
@ -1,67 +0,0 @@
|
||||
# Govarint
|
||||
|
||||
This project aims to provide a simple API for the performant encoding and decoding of 32 and 64 bit integers using a variety of algorithms.
|
||||
|
||||
[](https://www.flickr.com/photos/tsevis/8648521649/)
|
||||
|
||||
## Usage
|
||||
|
||||
Each integer encoding algorithm conforms to an encoding and decoding interface.
|
||||
The interfaces also specify the size of the unsigned integer, either 32 or 64 bits, and will be referred to as XX below.
|
||||
To create an encoder:
|
||||
|
||||
NewU32Base128Encoder(w io.Writer)
|
||||
NewU64Base128Encoder(w io.Writer)
|
||||
NewU32GroupVarintEncoder(w io.Writer)
|
||||
|
||||
For encoders, the only two commands are `PutUXX` and `Close`.
|
||||
`Close` must be called as some integer encoding algorithms write in multiples.
|
||||
|
||||
var buf bytes.Buffer
|
||||
enc := NewU32Base128Encoder(&buf)
|
||||
enc.PutU32(117)
|
||||
enc.PutU32(343)
|
||||
enc.Close()
|
||||
|
||||
To create a decoder:
|
||||
|
||||
NewU32Base128Decoder(r io.ByteReader)
|
||||
NewU64Base128Decoder(r io.ByteReader)
|
||||
NewU32GroupVarintDecoder(r io.ByteReader)
|
||||
|
||||
For decoders, the only command is `GetUXX`.
|
||||
`GetUXX` returns the value and any potential errors.
|
||||
When reading is complete, `GetUXX` will return an `EOF` (End Of File).
|
||||
|
||||
dec := NewU32Base128Decoder(&buf)
|
||||
x, err := dec.GetU32()
|
||||
|
||||
## Use Cases
|
||||
|
||||
Using fixed width integers, such as uint32 and uint64, usually waste large amounts of space, especially when encoding small values.
|
||||
Optimally, smaller numbers should take less space to represent.
|
||||
|
||||
Using integer encoding algorithms is especially common in specific applications, such as storing edge lists or indexes for search engines.
|
||||
In these situations, you have a sorted list of numbers that you want to keep as compactly as possible in memory.
|
||||
Additionally, by storing only the difference between the given number and the previous (delta encoding), the numbers are quite small, and thus compress well.
|
||||
|
||||
For an explicit example, the Web Data Commons Hyperlink Graph contains 128 billion edges linking page A to page B, where each page is represented by a 32 bit integer.
|
||||
By converting all these edges to 64 bit integers (32 | 32), sorting them, and then using delta encoding, memory usage can be reduced from 64 bits per edge down to only 9 bits per edge using the Base128 integer encoding algorithm.
|
||||
This figure improves even further if compressed using conventional compression algorithms (3 bits per edge).
|
||||
|
||||
## Encodings supported
|
||||
|
||||
`govarint` supports:
|
||||
|
||||
+ Base128 [32, 64] - each byte uses 7 bits for encoding the integer and 1 bit for indicating if the integer requires another byte
|
||||
+ Group Varint [32] - integers are encoded in blocks of four - one byte encodes the size of the following four integers, then the values of the four integers follows
|
||||
|
||||
Group Varint consistently beats Base128 in decompression speed but Base128 may offer improved compression ratios depending on the distribution of the supplied integers.
|
||||
|
||||
## Tests
|
||||
|
||||
go test -v -bench=.
|
||||
|
||||
## License
|
||||
|
||||
MIT License, as per `LICENSE`
|
||||
37
vendor/github.com/Unknwon/cae/README.md
generated
vendored
37
vendor/github.com/Unknwon/cae/README.md
generated
vendored
@ -1,37 +0,0 @@
|
||||
Compression and Archive Extensions
|
||||
==================================
|
||||
|
||||
[](http://gowalker.org/github.com/Unknwon/cae)
|
||||
|
||||
[中文文档](README_ZH.md)
|
||||
|
||||
Package cae implements PHP-like Compression and Archive Extensions.
|
||||
|
||||
But this package has some modifications depends on Go-style.
|
||||
|
||||
Reference: [PHP:Compression and Archive Extensions](http://www.php.net/manual/en/refs.compression.php).
|
||||
|
||||
Code Convention: based on [Go Code Convention](https://github.com/Unknwon/go-code-convention).
|
||||
|
||||
### Implementations
|
||||
|
||||
Package `zip`([Go Walker](http://gowalker.org/github.com/Unknwon/cae/zip)) and `tz`([Go Walker](http://gowalker.org/github.com/Unknwon/cae/tz)) both enable you to transparently read or write ZIP/TAR.GZ compressed archives and the files inside them.
|
||||
|
||||
- Features:
|
||||
- Add file or directory from everywhere to archive, no one-to-one limitation.
|
||||
- Extract part of entries, not all at once.
|
||||
- Stream data directly into `io.Writer` without any file system storage.
|
||||
|
||||
### Test cases and Coverage
|
||||
|
||||
All subpackages use [GoConvey](http://goconvey.co/) to write test cases, and coverage is more than 80 percent.
|
||||
|
||||
### Use cases
|
||||
|
||||
- [Gogs](https://github.com/gogits/gogs): self hosted Git service in the Go Programming Language.
|
||||
- [GoBlog](https://github.com/fuxiaohei/GoBlog): personal blogging application.
|
||||
- [GoBuild](https://github.com/shxsun/gobuild/): online Go cross-platform compilation and download service.
|
||||
|
||||
## License
|
||||
|
||||
This project is under Apache v2 License. See the [LICENSE](LICENSE) file for the full license text.
|
||||
29
vendor/github.com/Unknwon/cae/README_ZH.md
generated
vendored
29
vendor/github.com/Unknwon/cae/README_ZH.md
generated
vendored
@ -1,29 +0,0 @@
|
||||
压缩与打包扩展
|
||||
=============
|
||||
|
||||
[](http://gowalker.org/github.com/Unknwon/cae)
|
||||
|
||||
包 cae 实现了 PHP 风格的压缩与打包扩展。
|
||||
|
||||
但本包依据 Go 语言的风格进行了一些修改。
|
||||
|
||||
引用：[PHP:Compression and Archive Extensions](http://www.php.net/manual/en/refs.compression.php)
|
||||
|
||||
编码规范：基于 [Go 编码规范](https://github.com/Unknwon/go-code-convention)
|
||||
|
||||
### 实现
|
||||
|
||||
包 `zip`([Go Walker](http://gowalker.org/github.com/Unknwon/cae/zip)) 和 `tz`([Go Walker](http://gowalker.org/github.com/Unknwon/cae/tz)) 都允许你轻易的读取或写入 ZIP/TAR.GZ 压缩档案和其内部文件。
|
||||
|
||||
- 特性：
|
||||
- 将任意位置的文件或目录加入档案，没有一对一的操作限制。
|
||||
- 只解压部分文件，而非一次性解压全部。
|
||||
- 将数据以流的形式直接写入 `io.Writer` 而不需经过文件系统的存储。
|
||||
|
||||
### 测试用例与覆盖率
|
||||
|
||||
所有子包均采用 [GoConvey](http://goconvey.co/) 来书写测试用例，覆盖率均超过 80%。
|
||||
|
||||
## 授权许可
|
||||
|
||||
本项目采用 Apache v2 开源授权许可证，完整的授权说明已放置在 [LICENSE](LICENSE) 文件中。
|
||||
1
vendor/github.com/Unknwon/cae/tz/testdata/test.lnk
generated
vendored
Symbolic link
1
vendor/github.com/Unknwon/cae/tz/testdata/test.lnk
generated
vendored
Symbolic link
@ -0,0 +1 @@
|
||||
test.zip
|
||||
1
vendor/github.com/Unknwon/cae/tz/testdata/testdir.lnk
generated
vendored
Symbolic link
1
vendor/github.com/Unknwon/cae/tz/testdata/testdir.lnk
generated
vendored
Symbolic link
@ -0,0 +1 @@
|
||||
testdir
|
||||
1
vendor/github.com/Unknwon/cae/zip/testdata/test.lnk
generated
vendored
Symbolic link
1
vendor/github.com/Unknwon/cae/zip/testdata/test.lnk
generated
vendored
Symbolic link
@ -0,0 +1 @@
|
||||
test.zip
|
||||
1
vendor/github.com/Unknwon/cae/zip/testdata/testdir.lnk
generated
vendored
Symbolic link
1
vendor/github.com/Unknwon/cae/zip/testdata/testdir.lnk
generated
vendored
Symbolic link
@ -0,0 +1 @@
|
||||
testdir
|
||||
20
vendor/github.com/Unknwon/com/README.md
generated
vendored
20
vendor/github.com/Unknwon/com/README.md
generated
vendored
@ -1,20 +0,0 @@
|
||||
Common Functions
|
||||
================
|
||||
|
||||
[](https://travis-ci.org/Unknwon/com) [](http://gowalker.org/github.com/Unknwon/com)
|
||||
|
||||
This is an open source project for commonly used functions for the Go programming language.
|
||||
|
||||
This package need >= **go 1.2**
|
||||
|
||||
Code Convention: based on [Go Code Convention](https://github.com/Unknwon/go-code-convention).
|
||||
|
||||
## Contribute
|
||||
|
||||
Your contribute is welcome, but you have to check following steps after you added some functions and commit them:
|
||||
|
||||
1. Make sure you wrote user-friendly comments for **all functions** .
|
||||
2. Make sure you wrote test cases with any possible condition for **all functions** in file `*_test.go`.
|
||||
3. Make sure you wrote benchmarks for **all functions** in file `*_test.go`.
|
||||
4. Make sure you wrote useful examples for **all functions** in file `example_test.go`.
|
||||
5. Make sure you ran `go test` and got **PASS** .
|
||||
12
vendor/github.com/Unknwon/i18n/Makefile
generated
vendored
12
vendor/github.com/Unknwon/i18n/Makefile
generated
vendored
@ -1,12 +0,0 @@
|
||||
.PHONY: build test bench vet
|
||||
|
||||
build: vet bench
|
||||
|
||||
test:
|
||||
go test -v -cover
|
||||
|
||||
bench:
|
||||
go test -v -cover -test.bench=. -test.benchmem
|
||||
|
||||
vet:
|
||||
go vet
|
||||
136
vendor/github.com/Unknwon/i18n/README.md
generated
vendored
136
vendor/github.com/Unknwon/i18n/README.md
generated
vendored
@ -1,136 +0,0 @@
|
||||
i18n [](https://godoc.org/github.com/Unknwon/i18n) [](https://sourcegraph.com/github.com/Unknwon/i18n?badge)
|
||||
====
|
||||
|
||||
Package i18n is for app Internationalization and Localization.
|
||||
|
||||
## Introduction
|
||||
|
||||
This package provides multiple-language options to improve user experience. Sites like [Go Walker](http://gowalker.org) and [gogs.io](http://gogs.io) are using this module to implement Chinese and English user interfaces.
|
||||
|
||||
You can use following command to install this module:
|
||||
|
||||
go get github.com/Unknwon/i18n
|
||||
|
||||
## Usage
|
||||
|
||||
First of all, you have to import this package:
|
||||
|
||||
```go
|
||||
import "github.com/Unknwon/i18n"
|
||||
```
|
||||
|
||||
The format of locale files is very like INI format configuration file, which is basically key-value pairs. But this module has some improvements. Every language corresponding to a locale file, for example, under `conf/locale` folder of [gogsweb](https://github.com/gogits/gogsweb/tree/master/conf/locale), there are two files called `locale_en-US.ini` and `locale_zh-CN.ini`.
|
||||
|
||||
The name and extensions of locale files can be anything, but we strongly recommend you to follow the style of gogsweb.
|
||||
|
||||
## Minimal example
|
||||
|
||||
Here are two simplest locale file examples:
|
||||
|
||||
File `locale_en-US.ini`:
|
||||
|
||||
```ini
|
||||
hi = hello, %s
|
||||
bye = goodbye
|
||||
```
|
||||
|
||||
File `locale_zh-CN.ini`:
|
||||
|
||||
```ini
|
||||
hi = 您好，%s
|
||||
bye = 再见
|
||||
```
|
||||
|
||||
### Do Translation
|
||||
|
||||
There are two ways to do translation depends on which way is the best fit for your application or framework.
|
||||
|
||||
Directly use package function to translate:
|
||||
|
||||
```go
|
||||
i18n.Tr("en-US", "hi", "Unknwon")
|
||||
i18n.Tr("en-US", "bye")
|
||||
```
|
||||
|
||||
Or create a struct and embed it:
|
||||
|
||||
```go
|
||||
type MyController struct{
|
||||
// ...other fields
|
||||
i18n.Locale
|
||||
}
|
||||
|
||||
//...
|
||||
|
||||
func ... {
|
||||
c := &MyController{
|
||||
Locale: i18n.Locale{"en-US"},
|
||||
}
|
||||
_ = c.Tr("hi", "Unknwon")
|
||||
_ = c.Tr("bye")
|
||||
}
|
||||
```
|
||||
|
||||
Code above will produce correspondingly:
|
||||
|
||||
- English `en-US`：`hello, Unknwon`, `goodbye`
|
||||
- Chinese `zh-CN`：`您好，Unknwon`, `再见`
|
||||
|
||||
## Section
|
||||
|
||||
For different pages, one key may map to different values. Therefore, i18n module also uses the section feature of INI format configuration to achieve section.
|
||||
|
||||
For example, the key name is `about`, and we want to show `About` in the home page and `About Us` in about page. Then you can do following:
|
||||
|
||||
Content in locale file:
|
||||
|
||||
```ini
|
||||
about = About
|
||||
|
||||
[about]
|
||||
about = About Us
|
||||
```
|
||||
|
||||
Get `about` in home page:
|
||||
|
||||
```go
|
||||
i18n.Tr("en-US", "about")
|
||||
```
|
||||
|
||||
Get `about` in about page:
|
||||
|
||||
```go
|
||||
i18n.Tr("en-US", "about.about")
|
||||
```
|
||||
|
||||
### Ambiguity
|
||||
|
||||
Because dot `.` is sign of section in both [INI parser](https://github.com/go-ini/ini) and locale files, so when your key name contains `.` will cause ambiguity. At this point, you just need to add one more `.` in front of the key.
|
||||
|
||||
For example, the key name is `about.`, then we can use:
|
||||
|
||||
```go
|
||||
i18n.Tr("en-US", ".about.")
|
||||
```
|
||||
|
||||
to get expect result.
|
||||
|
||||
## Helper tool
|
||||
|
||||
Module i18n provides a command line helper tool beei18n for simplify steps of your development. You can install it as follows:
|
||||
|
||||
go get github.com/Unknwon/i18n/ui18n
|
||||
|
||||
### Sync locale files
|
||||
|
||||
Command `sync` allows you use a exist local file as the template to create or sync other locale files:
|
||||
|
||||
ui18n sync srouce_file.ini other1.ini other2.ini
|
||||
|
||||
This command can operate 1 or more files in one command.
|
||||
|
||||
## More information
|
||||
|
||||
- The first locale you load to the module is considered as **default locale**.
|
||||
- When matching non-default locale and didn't find the string, i18n will have a second try on default locale.
|
||||
- If i18n still cannot find string in the default locale, raw string will be returned. For instance, when the string is `hi` and it does not exist in locale file, simply return `hi` as output.
|
||||
65
vendor/github.com/Unknwon/paginater/README.md
generated
vendored
65
vendor/github.com/Unknwon/paginater/README.md
generated
vendored
@ -1,65 +0,0 @@
|
||||
Paginater [](https://drone.io/github.com/Unknwon/paginater/latest) [](http://gocover.io/github.com/Unknwon/paginater)
|
||||
=========
|
||||
|
||||
Package paginater is a helper module for custom pagination calculation.
|
||||
|
||||
## Installation
|
||||
|
||||
go get github.com/Unknwon/paginater
|
||||
|
||||
## Getting Started
|
||||
|
||||
The following code shows an example of how to use paginater:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import "github.com/Unknwon/paginater"
|
||||
|
||||
func main() {
|
||||
// Arguments:
|
||||
// - Total number of rows
|
||||
// - Number of rows in one page
|
||||
// - Current page number
|
||||
// - Number of page links
|
||||
p := paginater.New(45, 10, 3, 3)
|
||||
|
||||
// Then use p as a template object named "Page" in "demo.html"
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
`demo.html`
|
||||
|
||||
```html
|
||||
{{if not .Page.IsFirst}}[First](1){{end}}
|
||||
{{if .Page.HasPrevious}}[Previous]({{.Page.Previous}}){{end}}
|
||||
|
||||
{{range .Page.Pages}}
|
||||
{{if eq .Num -1}}
|
||||
...
|
||||
{{else}}
|
||||
{{.Num}}{{if .IsCurrent}}(current){{end}}
|
||||
{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if .Page.HasNext}}[Next]({{.Page.Next}}){{end}}
|
||||
{{if not .Page.IsLast}}[Last]({{.Page.TotalPages}}){{end}}
|
||||
```
|
||||
|
||||
Possible output:
|
||||
|
||||
```
|
||||
[First](1) [Previous](2) ... 2 3(current) 4 ... [Next](4) [Last](5)
|
||||
```
|
||||
|
||||
As you may guess, if the `Page` value is `-1`, you should print `...` in the HTML as common practice.
|
||||
|
||||
## Getting Help
|
||||
|
||||
- [API Documentation](https://gowalker.org/github.com/Unknwon/paginater)
|
||||
- [File An Issue](https://github.com/Unknwon/paginater/issues/new)
|
||||
|
||||
## License
|
||||
|
||||
This project is under Apache v2 License. See the [LICENSE](LICENSE) file for the full license text.
|
||||
7
vendor/github.com/andybalholm/cascadia/README.md
generated
vendored
7
vendor/github.com/andybalholm/cascadia/README.md
generated
vendored
@ -1,7 +0,0 @@
|
||||
# cascadia
|
||||
|
||||
[](https://travis-ci.org/andybalholm/cascadia)
|
||||
|
||||
The Cascadia package implements CSS selectors for use with the parse trees produced by the html package.
|
||||
|
||||
To test CSS selectors without writing Go code, check out [cascadia](https://github.com/suntong/cascadia) the command line tool, a thin wrapper around this package.
|
||||
16
vendor/github.com/blevesearch/bleve/CONTRIBUTING.md
generated
vendored
16
vendor/github.com/blevesearch/bleve/CONTRIBUTING.md
generated
vendored
@ -1,16 +0,0 @@
|
||||
# Contributing to Bleve
|
||||
|
||||
We look forward to your contributions, but ask that you first review these guidelines.
|
||||
|
||||
### Sign the CLA
|
||||
|
||||
As Bleve is a Couchbase project we require contributors accept the [Couchbase Contributor License Agreement](http://review.couchbase.org/static/individual_agreement.html). To sign this agreement log into the Couchbase [code review tool](http://review.couchbase.org/). The Bleve project does not use this code review tool but it is still used to track acceptance of the contributor license agreements.
|
||||
|
||||
### Submitting a Pull Request
|
||||
|
||||
All types of contributions are welcome, but please keep the following in mind:
|
||||
|
||||
- If you're planning a large change, you should really discuss it in a github issue or on the google group first. This helps avoid duplicate effort and spending time on something that may not be merged.
|
||||
- Existing tests should continue to pass, new tests for the contribution are nice to have.
|
||||
- All code should have gone through `go fmt`
|
||||
- All code should pass `go vet`
|
||||
67
vendor/github.com/blevesearch/bleve/README.md
generated
vendored
67
vendor/github.com/blevesearch/bleve/README.md
generated
vendored
@ -1,67 +0,0 @@
|
||||
#  bleve
|
||||
|
||||
[](https://travis-ci.org/blevesearch/bleve) [](https://coveralls.io/github/blevesearch/bleve?branch=master) [](https://godoc.org/github.com/blevesearch/bleve)
|
||||
[](https://gitter.im/blevesearch/bleve?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
[](https://codebeat.co/projects/github-com-blevesearch-bleve)
|
||||
[](https://goreportcard.com/report/blevesearch/bleve)
|
||||
[](https://sourcegraph.com/github.com/blevesearch/bleve?badge) [](https://opensource.org/licenses/Apache-2.0)
|
||||
|
||||
modern text indexing in go - [blevesearch.com](http://www.blevesearch.com/)
|
||||
|
||||
Try out bleve live by [searching the bleve website](http://www.blevesearch.com/search/?q=bleve).
|
||||
|
||||
## Features
|
||||
|
||||
* Index any go data structure (including JSON)
|
||||
* Intelligent defaults backed up by powerful configuration
|
||||
* Supported field types:
|
||||
* Text, Numeric, Date
|
||||
* Supported query types:
|
||||
* Term, Phrase, Match, Match Phrase, Prefix
|
||||
* Conjunction, Disjunction, Boolean
|
||||
* Numeric Range, Date Range
|
||||
* Simple query [syntax](http://www.blevesearch.com/docs/Query-String-Query/) for human entry
|
||||
* tf-idf Scoring
|
||||
* Search result match highlighting
|
||||
* Supports Aggregating Facets:
|
||||
* Terms Facet
|
||||
* Numeric Range Facet
|
||||
* Date Range Facet
|
||||
|
||||
## Discussion
|
||||
|
||||
Discuss usage and development of bleve in the [google group](https://groups.google.com/forum/#!forum/bleve).
|
||||
|
||||
## Indexing
|
||||
|
||||
```go
|
||||
message := struct{
|
||||
Id string
|
||||
From string
|
||||
Body string
|
||||
}{
|
||||
Id: "example",
|
||||
From: "marty.schoch@gmail.com",
|
||||
Body: "bleve indexing is easy",
|
||||
}
|
||||
|
||||
mapping := bleve.NewIndexMapping()
|
||||
index, err := bleve.New("example.bleve", mapping)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
index.Index(message.Id, message)
|
||||
```
|
||||
|
||||
## Querying
|
||||
|
||||
```go
|
||||
index, _ := bleve.Open("example.bleve")
|
||||
query := bleve.NewQueryStringQuery("bleve")
|
||||
searchRequest := bleve.NewSearchRequest(query)
|
||||
searchResult, _ := index.Search(searchRequest)
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache License Version 2.0
|
||||
7
vendor/github.com/blevesearch/bleve/analysis/test_words.txt
generated
vendored
7
vendor/github.com/blevesearch/bleve/analysis/test_words.txt
generated
vendored
@ -1,7 +0,0 @@
|
||||
# full line comment
|
||||
marty
|
||||
steve # trailing comment
|
||||
| different format of comment
|
||||
dustin
|
||||
siri | different style trailing comment
|
||||
multiple words with different whitespace
|
||||
9
vendor/github.com/blevesearch/bleve/geo/README.md
generated
vendored
9
vendor/github.com/blevesearch/bleve/geo/README.md
generated
vendored
@ -1,9 +0,0 @@
|
||||
# geo support in bleve
|
||||
|
||||
First, all of this geo code is a Go adaptation of the [Lucene 5.3.2 sandbox geo support](https://lucene.apache.org/core/5_3_2/sandbox/org/apache/lucene/util/package-summary.html).
|
||||
|
||||
## Notes
|
||||
|
||||
- All of the APIs will use float64 for lon/lat values.
|
||||
- When describing a point in function arguments or return values, we always use the order lon, lat.
|
||||
- High level APIs will use TopLeft and BottomRight to describe bounding boxes. This may not map cleanly to min/max lon/lat when crossing the dateline. The lower level APIs will use min/max lon/lat and require the higher-level code to split boxes accordingly.
|
||||
367
vendor/github.com/blevesearch/bleve/index/scorch/README.md
generated
vendored
367
vendor/github.com/blevesearch/bleve/index/scorch/README.md
generated
vendored
File diff suppressed because it is too large
Load Diff
167
vendor/github.com/blevesearch/bleve/index/scorch/segment/zap/README.md
generated
vendored
167
vendor/github.com/blevesearch/bleve/index/scorch/segment/zap/README.md
generated
vendored
@ -1,167 +0,0 @@
|
||||
# zap file format
|
||||
|
||||
The file is written in the reverse order that we typically access data. This helps us write in one pass since later sections of the file require file offsets of things we've already written.
|
||||
|
||||
Current usage:
|
||||
|
||||
- mmap the entire file
|
||||
- crc-32 bytes and version are in fixed position at end of the file
|
||||
- reading remainder of footer could be version specific
|
||||
- remainder of footer gives us:
|
||||
- 3 important offsets (docValue , fields index and stored data index)
|
||||
- 2 important values (number of docs and chunk factor)
|
||||
- field data is processed once and memoized onto the heap so that we never have to go back to disk for it
|
||||
- access to stored data by doc number means first navigating to the stored data index, then accessing a fixed position offset into that slice, which gives us the actual address of the data. the first bytes of that section tell us the size of data so that we know where it ends.
|
||||
- access to all other indexed data follows the following pattern:
|
||||
- first know the field name -> convert to id
|
||||
- next navigate to term dictionary for that field
|
||||
- some operations stop here and do dictionary ops
|
||||
- next use dictionary to navigate to posting list for a specific term
|
||||
- walk posting list
|
||||
- if necessary, walk posting details as we go
|
||||
- if location info is desired, consult location bitmap to see if it is there
|
||||
|
||||
## stored fields section
|
||||
|
||||
- for each document
|
||||
- preparation phase:
|
||||
- produce a slice of metadata bytes and data bytes
|
||||
- produce these slices in field id order
|
||||
- field value is appended to the data slice
|
||||
- metadata slice is govarint encoded with the following values for each field value
|
||||
- field id (uint16)
|
||||
- field type (byte)
|
||||
- field value start offset in uncompressed data slice (uint64)
|
||||
- field value length (uint64)
|
||||
- field number of array positions (uint64)
|
||||
- one additional value for each array position (uint64)
|
||||
- compress the data slice using snappy
|
||||
- file writing phase:
|
||||
- remember the start offset for this document
|
||||
- write out meta data length (varint uint64)
|
||||
- write out compressed data length (varint uint64)
|
||||
- write out the metadata bytes
|
||||
- write out the compressed data bytes
|
||||
|
||||
## stored fields idx
|
||||
|
||||
- for each document
|
||||
- write start offset (remembered from previous section) of stored data (big endian uint64)
|
||||
|
||||
With this index and a known document number, we have direct access to all the stored field data.
|
||||
|
||||
## posting details (freq/norm) section
|
||||
|
||||
- for each posting list
|
||||
- produce a slice containing multiple consecutive chunks (each chunk is govarint stream)
|
||||
- produce a slice remembering offsets of where each chunk starts
|
||||
- preparation phase:
|
||||
- for each hit in the posting list
|
||||
- if this hit is in next chunk close out encoding of last chunk and record offset start of next
|
||||
- encode term frequency (uint64)
|
||||
- encode norm factor (float32)
|
||||
- file writing phase:
|
||||
- remember start position for this posting list details
|
||||
- write out number of chunks that follow (varint uint64)
|
||||
- write out length of each chunk (each a varint uint64)
|
||||
- write out the byte slice containing all the chunk data
|
||||
|
||||
If you know the doc number you're interested in, this format lets you jump to the correct chunk (docNum/chunkFactor) directly and then seek within that chunk until you find it.
|
||||
|
||||
## posting details (location) section
|
||||
|
||||
- for each posting list
|
||||
- produce a slice containing multiple consecutive chunks (each chunk is govarint stream)
|
||||
- produce a slice remembering offsets of where each chunk starts
|
||||
- preparation phase:
|
||||
- for each hit in the posting list
|
||||
- if this hit is in next chunk close out encoding of last chunk and record offset start of next
|
||||
- encode field (uint16)
|
||||
- encode field pos (uint64)
|
||||
- encode field start (uint64)
|
||||
- encode field end (uint64)
|
||||
- encode number of array positions to follow (uint64)
|
||||
- encode each array position (each uint64)
|
||||
- file writing phase:
|
||||
- remember start position for this posting list details
|
||||
- write out number of chunks that follow (varint uint64)
|
||||
- write out length of each chunk (each a varint uint64)
|
||||
- write out the byte slice containing all the chunk data
|
||||
|
||||
If you know the doc number you're interested in, this format lets you jump to the correct chunk (docNum/chunkFactor) directly and then seek within that chunk until you find it.
|
||||
|
||||
## bitmaps of hits with location info
|
||||
|
||||
- for each posting list
|
||||
- preparation phase:
|
||||
- encode roaring bitmap (inidicating which hits have location details indexed) posting list to bytes (so we know the length)
|
||||
- file writing phase:
|
||||
- remember the start position for this bitmap
|
||||
- write length of encoded roaring bitmap
|
||||
- write the serialized roaring bitmap data
|
||||
|
||||
## postings list section
|
||||
|
||||
- for each posting list
|
||||
- preparation phase:
|
||||
- encode roaring bitmap posting list to bytes (so we know the length)
|
||||
- file writing phase:
|
||||
- remember the start position for this posting list
|
||||
- write freq/norm details offset (remembered from previous, as varint uint64)
|
||||
- write location details offset (remembered from previous, as varint uint64)
|
||||
- write location bitmap offset (remembered from pervious, as varint uint64)
|
||||
- write length of encoded roaring bitmap
|
||||
- write the serialized roaring bitmap data
|
||||
|
||||
## dictionary
|
||||
|
||||
- for each field
|
||||
- preparation phase:
|
||||
- encode vellum FST with dictionary data pointing to file offset of posting list (remembered from previous)
|
||||
- file writing phase:
|
||||
- remember the start position of this persistDictionary
|
||||
- write length of vellum data (varint uint64)
|
||||
- write out vellum data
|
||||
|
||||
## fields section
|
||||
|
||||
- for each field
|
||||
- file writing phase:
|
||||
- remember start offset for each field
|
||||
- write dictionary address (remembered from previous) (varint uint64)
|
||||
- write length of field name (varint uint64)
|
||||
- write field name bytes
|
||||
|
||||
## fields idx
|
||||
|
||||
- for each field
|
||||
- file writing phase:
|
||||
- write big endian uint64 of start offset for each field
|
||||
|
||||
NOTE: currently we don't know or record the length of this fields index. Instead we rely on the fact that we know it immediately precedes a footer of known size.
|
||||
|
||||
## fields DocValue
|
||||
|
||||
- for each field
|
||||
- preparation phase:
|
||||
- produce a slice containing multiple consecutive chunks, where each chunk is composed of a meta section followed by compressed columnar field data
|
||||
- produce a slice remembering the length of each chunk
|
||||
- file writing phase:
|
||||
- remember the start position of this first field DocValue offset in the footer
|
||||
- write out number of chunks that follow (varint uint64)
|
||||
- write out length of each chunk (each a varint uint64)
|
||||
- write out the byte slice containing all the chunk data
|
||||
|
||||
NOTE: currently the meta header inside each chunk gives clue to the location offsets and size of the data pertaining to a given docID and any
|
||||
read operation leverage that meta information to extract the document specific data from the file.
|
||||
|
||||
## footer
|
||||
|
||||
- file writing phase
|
||||
- write number of docs (big endian uint64)
|
||||
- write stored field index location (big endian uint64)
|
||||
- write field index location (big endian uint64)
|
||||
- write field docValue location (big endian uint64)
|
||||
- write out chunk factor (big endian uint32)
|
||||
- write out version (big endian uint32)
|
||||
- write out file CRC of everything preceding this (big endian uint32)
|
||||
8
vendor/github.com/blevesearch/bleve/index/upsidedown/benchmark_all.sh
generated
vendored
8
vendor/github.com/blevesearch/bleve/index/upsidedown/benchmark_all.sh
generated
vendored
@ -1,8 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
BENCHMARKS=`grep "func Benchmark" *_test.go | sed 's/.*func //' | sed s/\(.*{//`
|
||||
|
||||
for BENCHMARK in $BENCHMARKS
|
||||
do
|
||||
go test -v -run=xxx -bench=^$BENCHMARK$ -benchtime=10s -tags 'forestdb leveldb' | grep -v ok | grep -v PASS
|
||||
done
|
||||
14
vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.proto
generated
vendored
14
vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.proto
generated
vendored
@ -1,14 +0,0 @@
|
||||
message BackIndexTermsEntry {
|
||||
required uint32 field = 1;
|
||||
repeated string terms = 2;
|
||||
}
|
||||
|
||||
message BackIndexStoreEntry {
|
||||
required uint32 field = 1;
|
||||
repeated uint64 arrayPositions = 2;
|
||||
}
|
||||
|
||||
message BackIndexRowValue {
|
||||
repeated BackIndexTermsEntry termsEntries = 1;
|
||||
repeated BackIndexStoreEntry storedEntries = 2;
|
||||
}
|
||||
2909
vendor/github.com/blevesearch/bleve/search/facet/benchmark_data.txt
generated
vendored
2909
vendor/github.com/blevesearch/bleve/search/facet/benchmark_data.txt
generated
vendored
File diff suppressed because it is too large
Load Diff
328
vendor/github.com/blevesearch/bleve/search/query/query_string.y
generated
vendored
328
vendor/github.com/blevesearch/bleve/search/query/query_string.y
generated
vendored
File diff suppressed because it is too large
Load Diff
118
vendor/github.com/blevesearch/go-porterstemmer/README.md
generated
vendored
118
vendor/github.com/blevesearch/go-porterstemmer/README.md
generated
vendored
@ -1,118 +0,0 @@
|
||||
# This fork...
|
||||
|
||||
I'm maintaining this fork because the original author was not replying to issues or pull requests. For now I plan on maintaining this fork as necessary.
|
||||
|
||||
## Status
|
||||
|
||||
[](https://travis-ci.org/blevesearch/go-porterstemmer)
|
||||
|
||||
[](https://coveralls.io/r/blevesearch/go-porterstemmer?branch=HEAD)
|
||||
|
||||
# Go Porter Stemmer
|
||||
|
||||
A native Go clean room implementation of the Porter Stemming Algorithm.
|
||||
|
||||
This algorithm is of interest to people doing Machine Learning or
|
||||
Natural Language Processing (NLP).
|
||||
|
||||
This is NOT a port. This is a native Go implementation from the human-readable
|
||||
description of the algorithm.
|
||||
|
||||
I've tried to make it (more) efficient by NOT internally using string's, but
|
||||
instead internally using []rune's and using the same (array) buffer used by
|
||||
the []rune slice (and sub-slices) at all steps of the algorithm.
|
||||
|
||||
For Porter Stemmer algorithm, see:
|
||||
|
||||
http://tartarus.org/martin/PorterStemmer/def.txt (URL #1)
|
||||
|
||||
http://tartarus.org/martin/PorterStemmer/ (URL #2)
|
||||
|
||||
# Departures
|
||||
|
||||
Also, since when I initially implemented it, it failed the tests at...
|
||||
|
||||
http://tartarus.org/martin/PorterStemmer/voc.txt (URL #3)
|
||||
|
||||
http://tartarus.org/martin/PorterStemmer/output.txt (URL #4)
|
||||
|
||||
... after reading the human-readble text over and over again to try to figure out
|
||||
what the error I made was (and doing all sorts of things to debug it) I came to the
|
||||
conclusion that the some of these tests were wrong according to the human-readable
|
||||
description of the algorithm.
|
||||
|
||||
This led me to wonder if maybe other people's code that was passing these tests had
|
||||
rules that were not in the human-readable description. Which led me to look at the source
|
||||
code here...
|
||||
|
||||
http://tartarus.org/martin/PorterStemmer/c.txt (URL #5)
|
||||
|
||||
... When I looked there I noticed that there are some items marked as a "DEPARTURE",
|
||||
which differ from the original algorithm. (There are 2 of these.)
|
||||
|
||||
I implemented these departures, and the tests at URL #3 and URL #4 all passed.
|
||||
|
||||
## Usage
|
||||
|
||||
To use this Golang library, use with something like:
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/reiver/go-porterstemmer"
|
||||
)
|
||||
|
||||
func main() {
|
||||
|
||||
word := "Waxes"
|
||||
|
||||
stem := porterstemmer.StemString(word)
|
||||
|
||||
fmt.Printf("The word [%s] has the stem [%s].\n", word, stem)
|
||||
}
|
||||
|
||||
Alternatively, if you want to be a bit more efficient, use []rune slices instead, with code like:
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/reiver/go-porterstemmer"
|
||||
)
|
||||
|
||||
func main() {
|
||||
|
||||
word := []rune("Waxes")
|
||||
|
||||
stem := porterstemmer.Stem(word)
|
||||
|
||||
fmt.Printf("The word [%s] has the stem [%s].\n", string(word), string(stem))
|
||||
}
|
||||
|
||||
Although NOTE that the above code may modify original slice (named "word" in the example) as a side
|
||||
effect, for efficiency reasons. And that the slice named "stem" in the example above may be a
|
||||
sub-slice of the slice named "word".
|
||||
|
||||
Also alternatively, if you already know that your word is already lowercase (and you don't need
|
||||
this library to lowercase your word for you) you can instead use code like:
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/reiver/go-porterstemmer"
|
||||
)
|
||||
|
||||
func main() {
|
||||
|
||||
word := []rune("waxes")
|
||||
|
||||
stem := porterstemmer.StemWithoutLowerCasing(word)
|
||||
|
||||
fmt.Printf("The word [%s] has the stem [%s].\n", string(word), string(stem))
|
||||
}
|
||||
|
||||
Again NOTE (like with the previous example) that the above code may modify original slice (named
|
||||
"word" in the example) as a side effect, for efficiency reasons. And that the slice named "stem"
|
||||
in the example above may be a sub-slice of the slice named "word".
|
||||
92
vendor/github.com/blevesearch/segment/README.md
generated
vendored
92
vendor/github.com/blevesearch/segment/README.md
generated
vendored
@ -1,92 +0,0 @@
|
||||
# segment
|
||||
|
||||
A Go library for performing Unicode Text Segmentation
|
||||
as described in [Unicode Standard Annex #29](http://www.unicode.org/reports/tr29/)
|
||||
|
||||
## Features
|
||||
|
||||
* Currently only segmentation at Word Boundaries is supported.
|
||||
|
||||
## License
|
||||
|
||||
Apache License Version 2.0
|
||||
|
||||
## Usage
|
||||
|
||||
The functionality is exposed in two ways:
|
||||
|
||||
1. You can use a bufio.Scanner with the SplitWords implementation of SplitFunc.
|
||||
The SplitWords function will identify the appropriate word boundaries in the input
|
||||
text and the Scanner will return tokens at the appropriate place.
|
||||
|
||||
scanner := bufio.NewScanner(...)
|
||||
scanner.Split(segment.SplitWords)
|
||||
for scanner.Scan() {
|
||||
tokenBytes := scanner.Bytes()
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
2. Sometimes you would also like information returned about the type of token.
|
||||
To do this we have introduce a new type named Segmenter. It works just like Scanner
|
||||
but additionally a token type is returned.
|
||||
|
||||
segmenter := segment.NewWordSegmenter(...)
|
||||
for segmenter.Segment() {
|
||||
tokenBytes := segmenter.Bytes())
|
||||
tokenType := segmenter.Type()
|
||||
}
|
||||
if err := segmenter.Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
## Choosing Implementation
|
||||
|
||||
By default segment does NOT use the fastest runtime implementation. The reason is that it adds approximately 5s to compilation time and may require more than 1GB of ram on the machine performing compilation.
|
||||
|
||||
However, you can choose to build with the fastest runtime implementation by passing the build tag as follows:
|
||||
|
||||
-tags 'prod'
|
||||
|
||||
## Generating Code
|
||||
|
||||
Several components in this package are generated.
|
||||
|
||||
1. Several Ragel rules files are generated from Unicode properties files.
|
||||
2. Ragel machine is generated from the Ragel rules.
|
||||
3. Test tables are generated from the Unicode test files.
|
||||
|
||||
All of these can be generated by running:
|
||||
|
||||
go generate
|
||||
|
||||
## Fuzzing
|
||||
|
||||
There is support for fuzzing the segment library with [go-fuzz](https://github.com/dvyukov/go-fuzz).
|
||||
|
||||
1. Install go-fuzz if you haven't already:
|
||||
|
||||
go get github.com/dvyukov/go-fuzz/go-fuzz
|
||||
go get github.com/dvyukov/go-fuzz/go-fuzz-build
|
||||
|
||||
2. Build the package with go-fuzz:
|
||||
|
||||
go-fuzz-build github.com/blevesearch/segment
|
||||
|
||||
3. Convert the Unicode provided test cases into the initial corpus for go-fuzz:
|
||||
|
||||
go test -v -run=TestGenerateWordSegmentFuzz -tags gofuzz_generate
|
||||
|
||||
4. Run go-fuzz:
|
||||
|
||||
go-fuzz -bin=segment-fuzz.zip -workdir=workdir
|
||||
|
||||
## Status
|
||||
|
||||
|
||||
[](https://travis-ci.org/blevesearch/segment)
|
||||
|
||||
[](https://coveralls.io/r/blevesearch/segment?branch=master)
|
||||
|
||||
[](https://godoc.org/github.com/blevesearch/segment)
|
||||
285
vendor/github.com/blevesearch/segment/segment_words.rl
generated
vendored
285
vendor/github.com/blevesearch/segment/segment_words.rl
generated
vendored
@ -1,285 +0,0 @@
|
||||
// Copyright (c) 2015 Couchbase, Inc.
|
||||
// Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
|
||||
// except in compliance with the License. You may obtain a copy of the License at
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
// Unless required by applicable law or agreed to in writing, software distributed under the
|
||||
// License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
|
||||
// either express or implied. See the License for the specific language governing permissions
|
||||
// and limitations under the License.
|
||||
|
||||
// +build BUILDTAGS
|
||||
|
||||
package segment
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
var RagelFlags = "RAGELFLAGS"
|
||||
|
||||
var ParseError = fmt.Errorf("unicode word segmentation parse error")
|
||||
|
||||
// Word Types
|
||||
const (
|
||||
None = iota
|
||||
Number
|
||||
Letter
|
||||
Kana
|
||||
Ideo
|
||||
)
|
||||
|
||||
%%{
|
||||
machine s;
|
||||
write data;
|
||||
}%%
|
||||
|
||||
func segmentWords(data []byte, maxTokens int, atEOF bool, val [][]byte, types []int) ([][]byte, []int, int, error) {
|
||||
cs, p, pe := 0, 0, len(data)
|
||||
cap := maxTokens
|
||||
if cap < 0 {
|
||||
cap = 1000
|
||||
}
|
||||
if val == nil {
|
||||
val = make([][]byte, 0, cap)
|
||||
}
|
||||
if types == nil {
|
||||
types = make([]int, 0, cap)
|
||||
}
|
||||
|
||||
// added for scanner
|
||||
ts := 0
|
||||
te := 0
|
||||
act := 0
|
||||
eof := pe
|
||||
_ = ts // compiler not happy
|
||||
_ = te
|
||||
_ = act
|
||||
|
||||
// our state
|
||||
startPos := 0
|
||||
endPos := 0
|
||||
totalConsumed := 0
|
||||
%%{
|
||||
|
||||
include SCRIPTS "ragel/uscript.rl";
|
||||
include WB "ragel/uwb.rl";
|
||||
|
||||
action startToken {
|
||||
startPos = p
|
||||
}
|
||||
|
||||
action endToken {
|
||||
endPos = p
|
||||
}
|
||||
|
||||
action finishNumericToken {
|
||||
if !atEOF {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
|
||||
val = append(val, data[startPos:endPos+1])
|
||||
types = append(types, Number)
|
||||
totalConsumed = endPos+1
|
||||
if maxTokens > 0 && len(val) >= maxTokens {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
}
|
||||
|
||||
action finishHangulToken {
|
||||
if endPos+1 == pe && !atEOF {
|
||||
return val, types, totalConsumed, nil
|
||||
} else if dr, size := utf8.DecodeRune(data[endPos+1:]); dr == utf8.RuneError && size == 1 {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
|
||||
val = append(val, data[startPos:endPos+1])
|
||||
types = append(types, Letter)
|
||||
totalConsumed = endPos+1
|
||||
if maxTokens > 0 && len(val) >= maxTokens {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
}
|
||||
|
||||
action finishKatakanaToken {
|
||||
if endPos+1 == pe && !atEOF {
|
||||
return val, types, totalConsumed, nil
|
||||
} else if dr, size := utf8.DecodeRune(data[endPos+1:]); dr == utf8.RuneError && size == 1 {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
|
||||
val = append(val, data[startPos:endPos+1])
|
||||
types = append(types, Ideo)
|
||||
totalConsumed = endPos+1
|
||||
if maxTokens > 0 && len(val) >= maxTokens {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
}
|
||||
|
||||
action finishWordToken {
|
||||
if !atEOF {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
val = append(val, data[startPos:endPos+1])
|
||||
types = append(types, Letter)
|
||||
totalConsumed = endPos+1
|
||||
if maxTokens > 0 && len(val) >= maxTokens {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
}
|
||||
|
||||
action finishHanToken {
|
||||
if endPos+1 == pe && !atEOF {
|
||||
return val, types, totalConsumed, nil
|
||||
} else if dr, size := utf8.DecodeRune(data[endPos+1:]); dr == utf8.RuneError && size == 1 {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
|
||||
val = append(val, data[startPos:endPos+1])
|
||||
types = append(types, Ideo)
|
||||
totalConsumed = endPos+1
|
||||
if maxTokens > 0 && len(val) >= maxTokens {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
}
|
||||
|
||||
action finishHiraganaToken {
|
||||
if endPos+1 == pe && !atEOF {
|
||||
return val, types, totalConsumed, nil
|
||||
} else if dr, size := utf8.DecodeRune(data[endPos+1:]); dr == utf8.RuneError && size == 1 {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
|
||||
val = append(val, data[startPos:endPos+1])
|
||||
types = append(types, Ideo)
|
||||
totalConsumed = endPos+1
|
||||
if maxTokens > 0 && len(val) >= maxTokens {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
}
|
||||
|
||||
action finishNoneToken {
|
||||
lastPos := startPos
|
||||
for lastPos <= endPos {
|
||||
_, size := utf8.DecodeRune(data[lastPos:])
|
||||
lastPos += size
|
||||
}
|
||||
endPos = lastPos -1
|
||||
p = endPos
|
||||
|
||||
if endPos+1 == pe && !atEOF {
|
||||
return val, types, totalConsumed, nil
|
||||
} else if dr, size := utf8.DecodeRune(data[endPos+1:]); dr == utf8.RuneError && size == 1 {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
// otherwise, consume this as well
|
||||
val = append(val, data[startPos:endPos+1])
|
||||
types = append(types, None)
|
||||
totalConsumed = endPos+1
|
||||
if maxTokens > 0 && len(val) >= maxTokens {
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
}
|
||||
|
||||
HangulEx = Hangul ( Extend | Format )*;
|
||||
HebrewOrALetterEx = ( Hebrew_Letter | ALetter ) ( Extend | Format )*;
|
||||
NumericEx = Numeric ( Extend | Format )*;
|
||||
KatakanaEx = Katakana ( Extend | Format )*;
|
||||
MidLetterEx = ( MidLetter | MidNumLet | Single_Quote ) ( Extend | Format )*;
|
||||
MidNumericEx = ( MidNum | MidNumLet | Single_Quote ) ( Extend | Format )*;
|
||||
ExtendNumLetEx = ExtendNumLet ( Extend | Format )*;
|
||||
HanEx = Han ( Extend | Format )*;
|
||||
HiraganaEx = Hiragana ( Extend | Format )*;
|
||||
SingleQuoteEx = Single_Quote ( Extend | Format )*;
|
||||
DoubleQuoteEx = Double_Quote ( Extend | Format )*;
|
||||
HebrewLetterEx = Hebrew_Letter ( Extend | Format )*;
|
||||
RegionalIndicatorEx = Regional_Indicator ( Extend | Format )*;
|
||||
NLCRLF = Newline | CR | LF;
|
||||
OtherEx = ^(NLCRLF) ( Extend | Format )* ;
|
||||
|
||||
# UAX#29 WB8. Numeric × Numeric
|
||||
# WB11. Numeric (MidNum | MidNumLet | Single_Quote) × Numeric
|
||||
# WB12. Numeric × (MidNum | MidNumLet | Single_Quote) Numeric
|
||||
# WB13a. (ALetter | Hebrew_Letter | Numeric | Katakana | ExtendNumLet) × ExtendNumLet
|
||||
# WB13b. ExtendNumLet × (ALetter | Hebrew_Letter | Numeric | Katakana)
|
||||
#
|
||||
WordNumeric = ( ( ExtendNumLetEx )* NumericEx ( ( ( ExtendNumLetEx )* | MidNumericEx ) NumericEx )* ( ExtendNumLetEx )* ) >startToken @endToken;
|
||||
|
||||
# subset of the below for typing purposes only!
|
||||
WordHangul = ( HangulEx )+ >startToken @endToken;
|
||||
WordKatakana = ( KatakanaEx )+ >startToken @endToken;
|
||||
|
||||
# UAX#29 WB5. (ALetter | Hebrew_Letter) × (ALetter | Hebrew_Letter)
|
||||
# WB6. (ALetter | Hebrew_Letter) × (MidLetter | MidNumLet | Single_Quote) (ALetter | Hebrew_Letter)
|
||||
# WB7. (ALetter | Hebrew_Letter) (MidLetter | MidNumLet | Single_Quote) × (ALetter | Hebrew_Letter)
|
||||
# WB7a. Hebrew_Letter × Single_Quote
|
||||
# WB7b. Hebrew_Letter × Double_Quote Hebrew_Letter
|
||||
# WB7c. Hebrew_Letter Double_Quote × Hebrew_Letter
|
||||
# WB9. (ALetter | Hebrew_Letter) × Numeric
|
||||
# WB10. Numeric × (ALetter | Hebrew_Letter)
|
||||
# WB13. Katakana × Katakana
|
||||
# WB13a. (ALetter | Hebrew_Letter | Numeric | Katakana | ExtendNumLet) × ExtendNumLet
|
||||
# WB13b. ExtendNumLet × (ALetter | Hebrew_Letter | Numeric | Katakana)
|
||||
#
|
||||
# Marty -deviated here to allow for (ExtendNumLetEx x ExtendNumLetEx) part of 13a
|
||||
#
|
||||
Word = ( ( ExtendNumLetEx )* ( KatakanaEx ( ( ExtendNumLetEx )* KatakanaEx )*
|
||||
| ( HebrewLetterEx ( SingleQuoteEx | DoubleQuoteEx HebrewLetterEx )
|
||||
| NumericEx ( ( ( ExtendNumLetEx )* | MidNumericEx ) NumericEx )*
|
||||
| HebrewOrALetterEx ( ( ( ExtendNumLetEx )* | MidLetterEx ) HebrewOrALetterEx )*
|
||||
|ExtendNumLetEx
|
||||
)+
|
||||
)
|
||||
(
|
||||
( ExtendNumLetEx )+ ( KatakanaEx ( ( ExtendNumLetEx )* KatakanaEx )*
|
||||
| ( HebrewLetterEx ( SingleQuoteEx | DoubleQuoteEx HebrewLetterEx )
|
||||
| NumericEx ( ( ( ExtendNumLetEx )* | MidNumericEx ) NumericEx )*
|
||||
| HebrewOrALetterEx ( ( ( ExtendNumLetEx )* | MidLetterEx ) HebrewOrALetterEx )*
|
||||
)+
|
||||
)
|
||||
)* ExtendNumLetEx*) >startToken @endToken;
|
||||
|
||||
# UAX#29 WB14. Any ÷ Any
|
||||
WordHan = HanEx >startToken @endToken;
|
||||
WordHiragana = HiraganaEx >startToken @endToken;
|
||||
|
||||
WordExt = ( ( Extend | Format )* ) >startToken @endToken; # maybe plus not star
|
||||
|
||||
WordCRLF = (CR LF) >startToken @endToken;
|
||||
|
||||
WordCR = CR >startToken @endToken;
|
||||
|
||||
WordLF = LF >startToken @endToken;
|
||||
|
||||
WordNL = Newline >startToken @endToken;
|
||||
|
||||
WordRegional = (RegionalIndicatorEx+) >startToken @endToken;
|
||||
|
||||
Other = OtherEx >startToken @endToken;
|
||||
|
||||
main := |*
|
||||
WordNumeric => finishNumericToken;
|
||||
WordHangul => finishHangulToken;
|
||||
WordKatakana => finishKatakanaToken;
|
||||
Word => finishWordToken;
|
||||
WordHan => finishHanToken;
|
||||
WordHiragana => finishHiraganaToken;
|
||||
WordRegional =>finishNoneToken;
|
||||
WordCRLF => finishNoneToken;
|
||||
WordCR => finishNoneToken;
|
||||
WordLF => finishNoneToken;
|
||||
WordNL => finishNoneToken;
|
||||
WordExt => finishNoneToken;
|
||||
Other => finishNoneToken;
|
||||
*|;
|
||||
|
||||
write init;
|
||||
write exec;
|
||||
}%%
|
||||
|
||||
if cs < s_first_final {
|
||||
return val, types, totalConsumed, ParseError
|
||||
}
|
||||
|
||||
return val, types, totalConsumed, nil
|
||||
}
|
||||
18
vendor/github.com/boltdb/bolt/Makefile
generated
vendored
18
vendor/github.com/boltdb/bolt/Makefile
generated
vendored
@ -1,18 +0,0 @@
|
||||
BRANCH=`git rev-parse --abbrev-ref HEAD`
|
||||
COMMIT=`git rev-parse --short HEAD`
|
||||
GOLDFLAGS="-X main.branch $(BRANCH) -X main.commit $(COMMIT)"
|
||||
|
||||
default: build
|
||||
|
||||
race:
|
||||
@go test -v -race -test.run="TestSimulate_(100op|1000op)"
|
||||
|
||||
# go get github.com/kisielk/errcheck
|
||||
errcheck:
|
||||
@errcheck -ignorepkg=bytes -ignore=os:Remove github.com/boltdb/bolt
|
||||
|
||||
test:
|
||||
@go test -v -cover .
|
||||
@go test -v ./cmd/bolt
|
||||
|
||||
.PHONY: fmt test
|
||||
915
vendor/github.com/boltdb/bolt/README.md
generated
vendored
915
vendor/github.com/boltdb/bolt/README.md
generated
vendored
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user