InternetData Documentation

Usage

Every database endpoint is authenticated, so start with a key carrying the db.download scope:

using InternetData;

using var client = new InternetDataClient(Environment.GetEnvironmentVariable("INTERNETDATA_API_KEY")!);

foreach (var db in await client.Database.ListAsync())
{
    Console.WriteLine($"{db.Base}: {db.Standing}");   // bogon_ip: Licensed
}

The database calls hang off client.Database, which is where the sibling VPNDetection library keeps the same seven.

A license covers a database FAMILY, while a download names one of its versions, so the ids you pass to the other calls come from Versions:

var family = (await client.Database.ListAsync()).First(db => db.Standing == DatabaseStanding.Licensed);
var id = family.Versions[^1].Id;              // "bogon_ip_v1"
var formats = family.Versions[^1].Formats;    // [Csvgz, Mmdb]

Downloading a database

DownloadAsync streams to a path, so nothing bigger than a chunk is ever held in memory:

var written = await client.Database.DownloadAsync(id, DatabaseFormat.Csvgz, $"{id}.csv.gz");

The bytes land in a neighboring .part file that is renamed on completion, so a transfer that dies half way leaves nothing that reads as a whole database.

For a small database, take the bytes directly:

var bytes = await client.Database.DownloadBytesAsync("bogon_asn_v1", DatabaseFormat.Csvgz);

This holds the whole file in memory, and the catalog spans seven orders of magnitude, so check MetadataAsync first for anything you have not measured.

Handing the transfer to something else

The API answers a download with a 302 to a time-limited URL that carries its own authorization, so DownloadUrlAsync gives you a link you can pass to a downloader, a job queue or another machine without passing your key along with it:

var url = await client.Database.DownloadUrlAsync(id, DatabaseFormat.Mmdb);

The library follows nothing: you get the link, not the file. The link authorizes the START of a transfer, so one already running is not interrupted when it lapses.

Because a 302 is the answer, an HttpClient you supply yourself must not follow redirects, or it would fetch the whole database in place of the link. A client that does follow them is refused with a clear error rather than quietly downloading gigabytes.

Is today's build worth fetching?

MetadataAsync carries the build date, the row count and a size per format, without downloading anything:

var metadata = await client.Database.MetadataAsync(id);
Console.WriteLine($"{metadata.Updated} {metadata.Entries} rows, {metadata.Size["csvgz"]} bytes");
Console.WriteLine(metadata.UpdateFreq);          // "daily"

Verifying what arrived

var sums = await client.Database.ChecksumsAsync(id, DatabaseFormat.Csvgz);
Console.WriteLine(sums.Sha256);

What has been downloaded

Your organization's recent attempts, newest first. Refusals are listed too, which is what answers "it stopped working":

foreach (var attempt in await client.Database.DownloadsAsync(limit: 20))
{
    Console.WriteLine($"{attempt.Created:u} {attempt.DatasetId} {attempt.Outcome} {attempt.Bytes}");
}

Errors

Failures throw an InternetDataException carrying a Kind and a Retryable flag:

try
{
    await client.Database.MetadataAsync(id);
}
catch (InternetDataException e)
{
    Console.Error.WriteLine($"{e.Kind} {e.Retryable} {e.StatusCode} {e.Message}");
}

Kind is one of BadRequest, Unauthorized, Forbidden, RateLimited, QuotaExceeded, ServerError or Network. Message is the API's own reason code, so a license refusal reads NOT_LICENSED rather than 403.

Note that RateLimited and QuotaExceeded both arrive as HTTP 429 and are not the same thing. A rate limit is when the API faces extreme traffic bursts and so retrying later works; but a spent quota needs your allowance raised or the window to roll over. The library retries rate limits for you, but not if your quota is exceeded.

Dependency injection

The client takes an HttpClient, so it registers as a typed client and picks up your handler pipeline, pooling and resilience policies:

services.AddSingleton(new InternetDataClientOptions { ApiKey = builder.Configuration["InternetData:ApiKey"] });
services.AddHttpClient<InternetDataClient>()
    .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler { AllowAutoRedirect = false });

What you can see

ListAsync returns the catalog as your organization is entitled to see it.

A program running on the person's own machine can let them sign in with a browser and pick one of their API keys, instead of asking them to paste it:

using var client = new InternetDataClient();

var device = await client.Oauth.DeviceAuthorizationAsync(
    "your-client-id", new DeviceAuthorizationOptions { Scope = "account.read apikeys.read apikeys.reveal" });
Console.WriteLine($"Open {device.VerificationUri} and enter {device.UserCode}");

var token = await client.Oauth.PollDeviceTokenAsync("your-client-id", device);
if (token.Apikey is null)
{
    throw new InvalidOperationException("no API key came back: none was picked, or it can't be shown again");
}
using var keyed = new InternetDataClient(new InternetDataClientOptions { ApiKey = token.Apikey });

A denied sign-in throws OauthAccessDeniedException and a code that ran out OauthExpiredTokenException. Client IDs are issued on request from support@internetdata.io, and client.Oauth.RevokeAsync("your-client-id", token.RefreshToken) signs the machine out again.

.NET

Getting Started