Go

Getting Started

go get github.com/internetdata/sdk-go/v2

Requires Go 1.26 or newer. The module path ends in sdk-go, but the package it declares is internetdata:

import internetdata "github.com/internetdata/sdk-go/v2"

Usage

Every database published today is licensed, so start with an API key carrying the db.download scope. Create one in the console and pass it to New as an option:

client, err := internetdata.New(internetdata.WithAPIKey(os.Getenv("INTERNETDATA_API_KEY")))
if err != nil {
    log.Fatal(err)
}

databases, err := client.Database.List(ctx)
for _, db := range databases {
    fmt.Println(db.Base, db.Standing)   // bogon_ip licensed
}

Every database call hangs off client.Database, which is where the sibling VPNDetection library keeps the same seven calls.

The catalog

List returns one entry per database FAMILY, with your license beside it. A license covers the family, while a download names one of its versions, so the IDs you pass to everything else come from Versions:

for _, db := range databases {
    if db.Standing != internetdata.StandingLicensed {
        continue
    }
    for _, v := range db.Versions {
        fmt.Println(v.ID, v.Formats)    // vpn_ip_v1 [csvgz mmdb]
    }
}

Standing is licensed, expired or unlicensed, and LicenseType is what your license permits: evaluation, standard, redistribute, or nil when there is no license at all. Old versions are frozen rather than migrated, so both stay downloadable.

What is inside one, before you fetch it

Metadata carries the row count, the build date, the columns of each format and the byte size of each file, without downloading anything. Poll it to decide whether today's build is worth fetching, and read Size to budget a transfer before you start one:

meta, err := client.Database.Metadata(ctx, "vpn_ip_v1")
fmt.Println(meta.Updated, meta.Entries, meta.Size["mmdb"])

for _, column := range meta.Schema["csvgz"] {
    fmt.Println(column.Name, column.Type)
}

Downloading

DownloadFile writes one file to a path, streaming it straight to disk so that nothing bigger than a chunk is ever held in memory:

written, err := client.Database.DownloadFile(ctx, "vpn_ip_v1", internetdata.FormatMMDB, "./vpn_ip_v1.mmdb")
fmt.Printf("%d bytes\n", written)

The bytes land in a neighboring .part file that is renamed on completion, so a transfer that dies half way leaves no truncated file that reads as a whole database, and a failed refresh cannot destroy the copy already there.

Or stream it into a writer of your own, or take a small one as bytes:

written, err := client.Database.Download(ctx, "vpn_ip_v1", internetdata.FormatMMDB, w)
raw, err := client.Database.DownloadBytes(ctx, "bogon_ip_v1", internetdata.FormatCSVGZ)

DownloadBytes holds the whole file in memory, and the catalog spans five orders of magnitude, so use DownloadFile for anything you have not measured against Metadata's Size.

Format is a defined string type, so Format("zip") compiles. Anything the API does not publish is refused before the request leaves, as a bad_request naming what is allowed; Valid is the same check for a format you take from a flag or a config file:

if !internetdata.Format(fromFlag).Valid() {
    // csvgz or mmdb
}

Downloading a named build

Each download call fetches the build that is live unless you name one. A database.updated webhook names the build it announces and carries its sha256, so a receiver that downloads after the next publish asks for that build to get the file the hash is of:

written, err := client.Database.DownloadFile(ctx, "vpn_ip_v1", internetdata.FormatMMDB, "./vpn_ip_v1.mmdb",
    internetdata.Build(event.Data.Build))

A build is kept 7 days. Past that the call fails as bad_request with the message BUILD_GONE, and downloading without Build fetches the live one.

Downloading it yourself

The API answers a download with a redirect to a time-limited URL on object storage. DownloadURL hands you that URL rather than following it, so you can pass it to a downloader, a job queue or another machine:

url, err := client.Database.DownloadURL(ctx, "vpn_ip_v1", internetdata.FormatMMDB)

The link is presigned and so authorizes itself; it carries no API key of yours, and the library never sends your key to object storage. It authorizes the START of a transfer, so one already running is not interrupted when the link lapses.

Verifying a download

Checksums returns all four digests the exporter publishes for one file:

sums, err := client.Database.Checksums(ctx, "vpn_ip_v1", internetdata.FormatMMDB)
fmt.Println(sums.SHA256)

Download history

Downloads is your organization's recent attempts, newest first, refusals included: a denial is what answers "it stopped working", and its absence answers nothing.

history, err := client.Database.Downloads(ctx, 50)
for _, attempt := range history {
    fmt.Println(attempt.Created, attempt.DatasetID, attempt.Outcome)
}

Errors

Failures return an *internetdata.Error carrying a Kind, the API's own result code, and a Retryable flag:

_, err := client.Database.DownloadURL(ctx, "vpn_ip_v1", internetdata.FormatMMDB)
var apiErr *internetdata.Error
if errors.As(err, &apiErr) {
    fmt.Println(apiErr.Kind, apiErr.Message, apiErr.Retryable())
}

Kind is one of bad_request, unauthorized, forbidden, rate_limited, quota_exceeded, server_error or network. Message is the API's rc, which is what separates two refusals that share a status: NOT_LICENSED means buy it, LICENSE_EXPIRED means renew it, and IP_NOT_ALLOWED means the call came from an address the key or its organization does not allow, and all three are 403.

rate_limited is a 429 that carries Retry-After, and the library retries it for you. quota_exceeded is a 429 without one, which it never retries. InternetData's database API has no quota, so you won't see quota_exceeded from it.

Sign in with OAuth (device flow)

A program running on the person's own machine can let them sign in with a browser and pick one of their API keys, instead of asking them to paste it:

client, err := internetdata.New()
device, err := client.Oauth.DeviceAuthorization(ctx, "your-client-id",
    internetdata.DeviceAuthorizationOptions{Scope: "account.read apikeys.read apikeys.reveal"})
fmt.Printf("Open %s and enter %s\n", device.VerificationURI, device.UserCode)

token, err := client.Oauth.PollDeviceToken(ctx, "your-client-id", device)
if token.Apikey == nil {
    log.Fatal("no API key came back: none was picked, or it cannot be revealed")
}
keyed, err := internetdata.New(internetdata.WithAPIKey(*token.Apikey))

A denied sign-in satisfies errors.Is(err, internetdata.ErrOauthAccessDenied) and a code that ran out errors.Is(err, internetdata.ErrOauthExpiredToken), and client IDs are issued on request from support@internetdata.io. client.Oauth.Revoke(ctx, "your-client-id", *token.RefreshToken) signs the machine out again.

Sign in with OAuth (authorization code)

An app that can take a browser redirect signs the person in there instead, with a PKCE pair made for that one sign-in:

client, err := internetdata.New()
redirectURI := "http://127.0.0.1:8765/callback"
pkce := client.Oauth.CreatePkce()

authURL, err := client.Oauth.AuthorizationURL("your-client-id", redirectURI, pkce.Challenge,
    internetdata.AuthorizationURLOptions{Scope: "apikeys.use", State: "your-state"})
// Open authURL in the browser. Its redirect to redirectURI carries code and state.
token, err := client.Oauth.ExchangeAuthorizationCode(ctx, "your-client-id", code, pkce.Verifier, redirectURI)

Check that state came back as you sent it before you exchange code, which works once. The client ID can also be the https URL of a client metadata document your app serves, and such an app is never handed a key, so token.Apikey stays nil.

Register a client

An app with no client ID, and no client metadata document to use as one, registers itself, then signs in with the ClientID it gets back as the client ID for AuthorizationURL and ExchangeAuthorizationCode:

client, err := internetdata.New()
registration, err := client.Oauth.Register(ctx, []string{"http://127.0.0.1:8765/callback"},
    internetdata.RegisterOptions{ClientName: "Your App"})

Each redirect URI must be https on a public host name, or http on a loopback address, which then matches any port. Nothing is kept until a person approves the client, so an app with no client ID saved can simply register again.