InternetData Documentation

Getting Started

go get github.com/internetdata/sdk-go/v2

Requires Go 1.26 or newer. The module path ends in sdk-go, but the package it declares is internetdata:

import internetdata "github.com/internetdata/sdk-go/v2"

Usage

Every database published today is licensed, so start with an API key carrying the db.download scope. Create one in the console and pass it to New as an option:

client, err := internetdata.New(internetdata.WithAPIKey(os.Getenv("INTERNETDATA_API_KEY")))
if err != nil {
    log.Fatal(err)
}

databases, err := client.Database.List(ctx)
for _, db := range databases {
    fmt.Println(db.Base, db.Standing)   // bogon_ip licensed
}

Every database call hangs off client.Database, which is where the sibling VPNDetection library keeps the same seven calls.

List returns one entry per database FAMILY, with your license beside it. A license covers the family, while a download names one of its versions, so the ids you pass to everything else come from Versions:

for _, db := range databases {
    if db.Standing != internetdata.StandingLicensed {
        continue
    }
    for _, v := range db.Versions {
        fmt.Println(v.ID, v.Formats)    // vpn_ip_v1 [csvgz mmdb]
    }
}

Standing is licensed, expired or unlicensed, and LicenseType is what your license permits: evaluation, standard, redistribute, or nil when there is no license at all. Old versions are frozen rather than migrated, so both stay downloadable.

What is inside one, before you fetch it

Metadata carries the row count, the build date, the columns of each format and the byte size of each file, without downloading anything. Poll it to decide whether today's build is worth fetching, and read Size to budget a transfer before you start one:

meta, err := client.Database.Metadata(ctx, "vpn_ip_v1")
fmt.Println(meta.Updated, meta.Entries, meta.Size["mmdb"])

for _, column := range meta.Schema["csvgz"] {
    fmt.Println(column.Name, column.Type)
}

Downloading

DownloadFile writes one file to a path, streaming it straight to disk so that nothing bigger than a chunk is ever held in memory:

written, err := client.Database.DownloadFile(ctx, "vpn_ip_v1", internetdata.FormatMMDB, "./vpn_ip_v1.mmdb")
fmt.Printf("%d bytes\n", written)

The bytes land in a neighboring .part file that is renamed on completion, so a transfer that dies half way leaves no truncated file that reads as a whole database, and a failed refresh cannot destroy the copy already there.

Or stream it into a writer of your own, or take a small one as bytes:

written, err := client.Database.Download(ctx, "vpn_ip_v1", internetdata.FormatMMDB, w)
raw, err := client.Database.DownloadBytes(ctx, "bogon_ip_v1", internetdata.FormatCSVGZ)

DownloadBytes holds the whole file in memory, and the catalog spans five orders of magnitude, so use DownloadFile for anything you have not measured against Metadata's Size.

Format is a defined string type, so Format("zip") compiles. Anything the API does not publish is refused before the request leaves, as a bad_request naming what is allowed; Valid is the same check for a format you take from a flag or a config file:

if !internetdata.Format(fromFlag).Valid() {
    // csvgz or mmdb
}

Downloading it yourself

The API answers a download with a redirect to a time-limited URL on object storage. DownloadURL hands you that URL rather than following it, so you can pass it to a downloader, a job queue or another machine:

url, err := client.Database.DownloadURL(ctx, "vpn_ip_v1", internetdata.FormatMMDB)

The link is presigned and so authorizes itself; it carries no API key of yours, and the library never sends your key to object storage. It authorizes the START of a transfer, so one already running is not interrupted when the link lapses.

Verifying a download

Checksums returns all four digests the exporter publishes for one file:

sums, err := client.Database.Checksums(ctx, "vpn_ip_v1", internetdata.FormatMMDB)
fmt.Println(sums.SHA256)

Download history

Downloads is your organization's recent attempts, newest first, refusals included: a denial is what answers "it stopped working", and its absence answers nothing.

history, err := client.Database.Downloads(ctx, 50)
for _, attempt := range history {
    fmt.Println(attempt.Created, attempt.DatasetID, attempt.Outcome)
}

Errors

Failures return an *internetdata.Error carrying a Kind, the API's own result code, and a Retryable flag:

_, err := client.Database.DownloadURL(ctx, "vpn_ip_v1", internetdata.FormatMMDB)
var apiErr *internetdata.Error
if errors.As(err, &apiErr) {
    fmt.Println(apiErr.Kind, apiErr.Message, apiErr.Retryable())
}

Kind is one of bad_request, unauthorized, forbidden, rate_limited, quota_exceeded, server_error or network. Message is the API's rc, which is what separates two refusals that share a status: NOT_LICENSED means buy it and LICENSE_EXPIRED means renew it, and both are 403.

Note that rate_limited and quota_exceeded both arrive as HTTP 429 and are not the same thing. A rate limit is when the API faces extreme traffic bursts and so retrying later works; but a spent quota needs your allowance raised or the window to roll over. The library retries rate limits for you, but not if your quota is exceeded.

A program running on the person's own machine can let them sign in with a browser and pick one of their API keys, instead of asking them to paste it:

client, err := internetdata.New()
device, err := client.Oauth.DeviceAuthorization(ctx, "your-client-id",
    internetdata.DeviceAuthorizationOptions{Scope: "account.read apikeys.read apikeys.reveal"})
fmt.Printf("Open %s and enter %s\n", device.VerificationURI, device.UserCode)

token, err := client.Oauth.PollDeviceToken(ctx, "your-client-id", device)
if token.Apikey == nil {
    log.Fatal("no API key came back: none was picked, or it cannot be revealed")
}
keyed, err := internetdata.New(internetdata.WithAPIKey(*token.Apikey))

A denied sign-in satisfies errors.Is(err, internetdata.ErrOauthAccessDenied) and a code that ran out errors.Is(err, internetdata.ErrOauthExpiredToken), and client IDs are issued on request from support@internetdata.io. client.Oauth.Revoke(ctx, "your-client-id", *token.RefreshToken) signs the machine out again.

Go