.NET

Getting Started

dotnet add package InternetData

Targets .NET 8 and newer.

Usage

Every database endpoint is authenticated, so start with a key carrying the db.download scope:

using InternetData;

using var client = new InternetDataClient(Environment.GetEnvironmentVariable("INTERNETDATA_API_KEY")!);

foreach (var db in await client.Database.ListAsync())
{
    Console.WriteLine($"{db.Base}: {db.Standing}");   // bogon_ip: Licensed
}

The database calls hang off client.Database, which is where the sibling VPNDetection library keeps the same seven.

A license covers a database FAMILY, while a download names one of its versions, so the IDs you pass to the other calls come from Versions:

var family = (await client.Database.ListAsync()).First(db => db.Standing == Standing.Licensed);
var id = family.Versions[^1].Id;              // "bogon_ip_v1"
var formats = family.Versions[^1].Formats;    // [Csvgz, Mmdb]

Downloading a database

DownloadAsync streams to a path, so nothing bigger than a chunk is ever held in memory:

var written = await client.Database.DownloadAsync(id, DatabaseFormat.Csvgz, $"{id}.csv.gz");

The bytes land in a neighboring .part file that is renamed on completion, so a transfer that dies half way leaves nothing that reads as a whole database.

For a small database, take the bytes directly:

var bytes = await client.Database.DownloadBytesAsync("bogon_asn_v1", DatabaseFormat.Csvgz);

This holds the whole file in memory, and the catalog spans seven orders of magnitude, so check MetadataAsync first for anything you have not measured.

Handing the transfer to something else

The API answers a download with a 302 to a time-limited URL that carries its own authorization, so DownloadUrlAsync gives you a link you can pass to a downloader, a job queue or another machine without passing your key along with it:

var url = await client.Database.DownloadUrlAsync(id, DatabaseFormat.Mmdb);

The library follows nothing: you get the link, not the file. The link authorizes the START of a transfer, so one already running is not interrupted when it lapses.

Because a 302 is the answer, an HttpClient you supply yourself must not follow redirects, or it would fetch the whole database in place of the link. A client that does follow them is refused with a clear error rather than quietly downloading gigabytes.

Downloading a named build

Each download call fetches the build that is live unless you name one. A database.updated webhook names the build it announces and carries its sha256, so a receiver that downloads after the next publish asks for that build to get the file the hash is of:

await client.Database.DownloadAsync("vpn_ip_v1", DatabaseFormat.Mmdb, "./vpn_ip_v1.mmdb", build);

A build is kept 7 days. Past that the call throws BadRequest with the message BUILD_GONE, and downloading without a build fetches the live one.

Is today's build worth fetching?

MetadataAsync carries the build date, the row count and a size per format, without downloading anything:

var metadata = await client.Database.MetadataAsync(id);
Console.WriteLine($"{metadata.Updated} {metadata.Entries} rows, {metadata.Size["csvgz"]} bytes");
Console.WriteLine(metadata.UpdateFreq);          // "daily"

Verifying what arrived

var sums = await client.Database.ChecksumsAsync(id, DatabaseFormat.Csvgz);
Console.WriteLine(sums.Sha256);

What has been downloaded

Your organization's recent attempts, newest first. Refusals are listed too, which is what answers "it stopped working":

foreach (var attempt in await client.Database.DownloadsAsync(limit: 20))
{
    Console.WriteLine($"{attempt.Created:u} {attempt.DatasetId} {attempt.Outcome} {attempt.Bytes}");
}

Errors

Failures throw an InternetDataException carrying a Kind and a Retryable flag:

try
{
    await client.Database.MetadataAsync(id);
}
catch (InternetDataException e)
{
    Console.Error.WriteLine($"{e.Kind} {e.Retryable} {e.StatusCode} {e.Message}");
}

Kind is one of BadRequest, Unauthorized, Forbidden, RateLimited, QuotaExceeded, ServerError or Network. Message is the API's own reason code, so a license refusal reads NOT_LICENSED rather than 403.

RateLimited is a 429 that carries Retry-After, and the library retries it for you. QuotaExceeded is a 429 without one, which it never retries. InternetData's database API has no quota, so you won't see QuotaExceeded from it.

Dependency injection

The client takes an HttpClient, so it registers as a typed client and picks up your handler pipeline, pooling and resilience policies:

services.AddSingleton(new InternetDataClientOptions { ApiKey = builder.Configuration["InternetData:ApiKey"] });
services.AddHttpClient<InternetDataClient>()
    .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler { AllowAutoRedirect = false });

What you can see

ListAsync returns the catalog as your organization is entitled to see it.

Sign in with OAuth (device flow)

A program running on the person's own machine can let them sign in with a browser and pick one of their API keys, instead of asking them to paste it:

using var client = new InternetDataClient();

var device = await client.Oauth.DeviceAuthorizationAsync(
    "your-client-id", new DeviceAuthorizationOptions { Scope = "account.read apikeys.read apikeys.reveal" });
Console.WriteLine($"Open {device.VerificationUri} and enter {device.UserCode}");

var token = await client.Oauth.PollDeviceTokenAsync("your-client-id", device);
if (token.Apikey is null)
{
    throw new InvalidOperationException("no API key came back: none was picked, or it can't be shown again");
}
using var keyed = new InternetDataClient(new InternetDataClientOptions { ApiKey = token.Apikey });

A denied sign-in throws OauthAccessDeniedException and a code that ran out OauthExpiredTokenException. Client IDs are issued on request from support@internetdata.io, and client.Oauth.RevokeAsync("your-client-id", token.RefreshToken) signs the machine out again.

Sign in with OAuth (authorization code)

An app that can take a browser redirect signs the person in there instead, with a PKCE pair made for that one sign-in:

using var client = new InternetDataClient();
var redirectUri = "http://127.0.0.1:8765/callback";
var pkce = client.Oauth.CreatePkce();

var url = client.Oauth.AuthorizationUrl("your-client-id", redirectUri, pkce.Challenge,
    new AuthorizationUrlOptions { Scope = "apikeys.use", State = "your-state" });
// Open url in the browser. Its redirect to redirectUri carries code and state.
var token = await client.Oauth.ExchangeAuthorizationCodeAsync("your-client-id", code, pkce.Verifier, redirectUri);

Check that state came back as you sent it before you exchange code, which works once. The client ID can also be the https URL of a client metadata document your app serves, and such an app is never handed a key, so token.Apikey stays null.

Register a client

An app with no client ID, and no client metadata document to use as one, registers itself, then signs in with the ClientId it gets back as the client ID for AuthorizationUrl and ExchangeAuthorizationCodeAsync:

using var client = new InternetDataClient();
var registration = await client.Oauth.RegisterAsync(new[] { "http://127.0.0.1:8765/callback" },
    new RegisterOptions { ClientName = "Your App" });

Each redirect URI must be https on a public host name, or http on a loopback address, which then matches any port. Nothing is kept until a person approves the client, so an app with no client ID saved can simply register again.