Tutorial: Notion sync

This tutorial is created in order to provide help on creating complex integration app with dynamic data schema, non-primitive data synchronization (for example, files) and oauth2 authentication. The source code (node.js) can be found in official Fibery repository which contains the implementation of integrating Notion databases into Fibery databases. Demo databases can be found here.

App Configuration

Returns the description of the app and possible ways to be authenticated in Notion. Route in app.js

app.get(`/`, (req, res) => res.json(connector()));

connector.config.js:

const config = require(`./config`);
const ApiKeyAuthentication = {
  description: `Please provide notion authentication`,
  name: `Token`,
  id: `key`,
  fields: [
    {
      type: `password`,
      name: `Integration Token`,
      description: `Provide Notion API Integration Token`,
      id: `key`,
    },
    {
      type: `link`,
      value: `https://www.notion.so/help/create-integrations-with-the-notion-api`,
      description: `We need to have your Notion Integration Token to synchronize the data.`,
      id: `key-link`,
      name: `Read how to create integration, grant access and create token here...`,
    },
  ],
};
const OAuth2 = {
  id: 'oauth2',
  name: 'OAuth v2 Authentication',
  description: 'OAuth v2-based authentication and authorization for access to Notion',
  fields: [
    {
      title: 'callback_uri',
      description: 'OAuth post-auth redirect URI',
      type: 'oauth',
      id: 'callback_uri',
    },
  ],
};

const getAuthenticationStrategies = () => {
  return [OAuth2, ApiKeyAuthentication];
};

module.exports.connector = () => ({
  id: `notion-app`,
  name: `Notion`,
  version: config.version,
  website: `https://notion.com`,
  description: `More than a doc. Or a table. Customize Notion to work the way you do.`,
  authentication: getAuthenticationStrategies(),
  responsibleFor: {
    dataSynchronization: true,
  },
  sources: [],
});

As you see there are two authentication ways are defined:

OAuth2

Hardcoded "oauth2" should be used as id in case you would like to implement OAuth2 support in integration app.

Token Authentication

You may use special field type: "link" in order to provide url for external resource where the user can get more info. Use type:"password" for tokens or other text fields which need to be secured.

Token Authorization

The implementation of token authentication is the simplest way to implement. We always used it for testing and development since it is not required UI interaction. The request contains id of auth and user provided values. In our case it is key. Other fields are appended by system and can be ignored. Route (app.js):

app.post(`/validate`, (req, res) => promiseToResponse(res, notion.validate(_.get(req, `body.fields`) || req.body)));

Request Body:

{
  "id": "key",
  "fields": {
    "app": "620a3c9baec5dd25794fed7a",
    "auth": "key",
    "owner": "620a3c46cf7154924cf442cb",
    "key": "MY TOKEN",
    "enabled": true
  }
}

Notion call (the name of account is returned):

module.exports.validate = async (account) => {
  const client = getNotionClient(account);
  const me = await client.users.me();
  return {name: me.name}; // response should include the name of user account
};

OAuth 2

OAuth 2 is a bit more complex and requires several routes to be implemented. The POST /oauth2/v1/authorize endpoint performs the initial setup for OAuth version 2 accounts using Authorization Code grant type by generating redirect_uri based on received parameters.

Synchronizer configuration

This endpoint returns types which should be synced to Fibery databases. In Notion case it is the list of databases. Static user type is added. Check how the configuration response looks like for Notion Demo.

app.js (route)

app.post(`/api/v1/synchronizer/config`, (req, res) => {
  if (_.isEmpty(req.body.account)) {
    throw new Error(`account should be provided`);
  }
  promiseToResponse(res, notion.config(req.body));
});

notion.api.js

const getDatabases = async ({account, pageSize = 1000}) => {
  const client = getNotionClient(account);
  let hasNext = true;
  let start_cursor = null;
  const databases = [];
  while (hasNext) {
    const args = {
      page_size: pageSize,
      filter: {
        value: `database`,
        property: `object`,
      }
    };
    if (start_cursor) {
      args.start_cursor = start_cursor;
    }
    const {results, has_more, next_cursor} = await client.search(args);
    results.forEach((db) => databases.push(db));
    hasNext = has_more;
    start_cursor = next_cursor;
  }
  return databases;
};

const getDatabaseItem = (db) => {
  const name = _.get(db, `title[0].plain_text`, `Noname`).replace(/[^\w ]+/g, ``).trim();
  return {id: db.id, name};
};

module.exports.config = async ({account, pageSize}) => {
  const databases = await getDatabases({account, pageSize});
  const dbItems = databases.map((db) => getDatabaseItem(db)).concat({id: `user`, name: `User`});
  return {types: dbItems, filters: []};
};

Response example

{
  "types": [
    {
      "id": "f4642444-220c-439d-85d6-378ddff3d510",
      "name": "Features"
    },
    {
      "id": "3bd058e6-a71c-4e9a-8480-a76810ae38d3",
      "name": "Tasks"
    },
    {
      "id": "user",
      "name": "User"
    }
  ],
  "filters": []
}

Schema of synchronization

The schema which describes fields and relations should be provided for each sync type. Find full implementation here. It is not easy thing to implement since we are talking about dynamic data in Notion databases. app.js (schema route)

app.post(`/api/v1/synchronizer/schema`, (req, res) => promiseToResponse(res, notion.schema(req.body)));

notion.api.js

module.exports.schema = async ({account, types}) => {
  const databases = await getDatabases({account});
  const mapDatabasesById = _.keyBy(databases, `id`);
  const schema = {};
  types.forEach((id) => {
    if (id === `user`) {
      schema.user = userSchema;
      return;
    }
    const db = mapDatabasesById[id];
    if (_.isEmpty(db)) {
      throw new Error(`Database with id "${id}" is not found`);
    }
    schema[id] = createSchemaFromDatabase(db);
  });
  cleanRelationsDuplication(schema);
  return schema;
};

Request example:

{
  "account": {
    "_id": "620a4396aec5dd672c4fed83",
    "access_token": "USER-TOKEN",
    "app": "620a3c9baec5dd25794fed7a",
    "auth": "oauth2",
    "owner": "620a3c46cf7154924cf442cb",
    "enabled": true,
    "name": "Fibery Developer",
    "masterAccountId": null,
    "lastUpdatedOn": "2022-02-21T09:45:37.802Z"
  },
  "filter": {},
  "types": [
    "f4642444-220c-439d-85d6-378ddff3d510",
    "3bd058e6-a71c-4e9a-8480-a76810ae38d3",
    "user"
  ]
}

Response example:

{
  "f4642444-220c-439d-85d6-378ddff3d510": {
    "id": {
      "type": "id",
      "name": "Id"
    },
    "archived": {
      "type": "text",
      "name": "Archived",
      "subType": "boolean"
    },
    "created_time": {
      "type": "date",
      "name": "Created On"
    },
    "last_edited_time": {
      "type": "date",
      "name": "Last Edited On"
    },
    "__notion_link": {
      "type": "text",
      "name": "Notion Link",
      "subType": "url"
    },
    "related to tasks (column)": {
      "name": "Related to Tasks (Column) Ref",
      "type": "text",
      "relation": {
        "cardinality": "many-to-many",
        "targetFieldId": "id",
        "name": "Related to Tasks (Column)",
        "targetName": "Feature",
        "targetType": "3bd058e6-a71c-4e9a-8480-a76810ae38d3"
      }
    },
    "tags": {
      "name": "Tags",
      "type": "array[text]"
    },
    "due date": {
      "name": "Due Date",
      "type": "date"
    },
    "name": {
      "name": "Name",
      "type": "text"
    }
  },
  "3bd058e6-a71c-4e9a-8480-a76810ae38d3": {
    "id": {
      "type": "id",
      "name": "Id"
    },
    "archived": {
      "type": "text",
      "name": "Archived",
      "subType": "boolean"
    },
    "created_time": {
      "type": "date",
      "name": "Created On"
    },
    "last_edited_time": {
      "type": "date",
      "name": "Last Edited On"
    },
    "__notion_link": {
      "type": "text",
      "name": "Notion Link",
      "subType": "url"
    },
    "status": {
      "name": "Status",
      "type": "text"
    },
    "assignees": {
      "name": "Assignees Ref",
      "type": "array[text]",
      "relation": {
        "cardinality": "many-to-many",
        "targetType": "user",
        "targetFieldId": "id",
        "name": "Assignees",
        "targetName": "Tasks (Assignees Ref)"
      }
    },
    "specs": {
      "name": "Specs",
      "type": "array[text]",
      "subType": "file"
    },
    "link to site": {
      "name": "Link to site",
      "type": "text",
      "subType": "url"
    },
    "name": {
      "name": "Name",
      "type": "text"
    }
  },
  "user": {
    "id": {
      "type": "id",
      "name": "Id",
      "path": "id"
    },
    "name": {
      "type": "text",
      "name": "Name",
      "path": "name"
    },
    "type": {
      "type": "text",
      "name": "Type",
      "path": "type"
    },
    "email": {
      "type": "text",
      "name": "Email",
      "subType": "email"
    }
  }
}

It can be noticed that almost any field from Notion database can be mapped into Fibery field using subType attribute. Relations can be mapped as well. Rich text can be sent as html or md by defining corresponding type="text" and subType="md" or "html". Note: Relation between databases(types) should be declared only once. Double declarations for relations will lead to duplication of relations in Fibery databases. We implemented the function cleanRelationsDuplication in order to remove redundant relation declarations from schema fields. Files field mapping:

"specs": {
  "name": "Specs",
  "type": "array[text]",
  "subType": "file"
}

Data route

Notion supports paged output, so it is handy to fetch data page by page. The response should include pagination node with hasNext equals to true or false and nextPageConfig (next page configuration) which will be included with the future request as pagination. You may notice that we have included schema into nextPageConfig (pagination config). It is not required and it is done as an optimization in order to save some between pages fetching on schema resolving. In other words the pagination can be used as a context cache between page calls. app.js

app.post(`/api/v1/synchronizer/data`, (req, res) => promiseToResponse(res, notion.data(req.body)));

notion.api.js (paging support)

const getValue = (row, {path, arrayPath, subPath = ``}) => {
  let v = null;
  const paths = _.isArray(path) ? path : [path];
  paths.forEach((p) => {
    if (!_.isUndefined(v) && !_.isNull(v)) {
      return;
    }
    v = _.get(row, p);
  });

  if (!_.isEmpty(subPath) && _.isObject(v)) {
    return getValue(v, {path: subPath});
  }

  if (!_.isEmpty(arrayPath) && _.isArray(v)) {
    return v.map((element) => getValue(element, {path: arrayPath}));
  }

  if (_.isObject(v)) {
    if (v.start) {
      return v.start;
    }
    if (v.end) {
      return v.end;
    }
    if (v.type) {
      return v[v.type];
    }
    return JSON.stringify(v);
  }

  return v;
};

const processItem = ({schema, item}) => {
  const r = {};
  _.keys(schema).forEach((id) => {
    const schemaValue = schema[id];
    r[id] = getValue(item, schemaValue);
  });
  return r;
};

const resolveSchema = async ({pagination, client, requestedType}) => {
  if (pagination && pagination.schema) {
    return pagination.schema;
  }
  if (requestedType === `user`) {
    return userSchema;
  }
  return createSchemaFromDatabase(await client.databases.retrieve({database_id: requestedType}));
};

const createArgs = ({pageSize, pagination, requestedType}) => {
  const args = {
    page_size: pageSize,
  };
  if (!_.isEmpty(pagination) && !_.isEmpty(pagination.start_cursor)) {
    args.start_cursor = pagination.start_cursor;
  }
  if (requestedType !== `user`) {
    args.database_id = requestedType;
  }
  return args;
};

module.exports.data = async ({account, requestedType, pageSize = 1000, pagination}) => {
  const client = getNotionClient(account);
  const schema = await resolveSchema({pagination, client, requestedType});
  const args = createArgs({pageSize, pagination, requestedType});
  const data = requestedType !== `user`
    ? await client.databases.query(args)
    : await client.users.list(args);
  const {results, next_cursor, has_more} = data;
  return {
    items: results.map((item) => processItem({account, schema, item})),
    "pagination": {
      "hasNext": has_more,
      "nextPageConfig": {
        start_cursor: next_cursor,
        schema: has_more ? schema : null,
      },
    },
  };
};

Request example:

{
  "filter": {},
  "types": [
    "f4642444-220c-439d-85d6-378ddff3d510",
    "3bd058e6-a71c-4e9a-8480-a76810ae38d3",
    "user"
  ],
  "requestedType": "3bd058e6-a71c-4e9a-8480-a76810ae38d3",
  "account": {
    "_id": "620a4396aec5dd672c4fed83",
    "access_token": "USER-TOKEN",
    "app": "620a3c9baec5dd25794fed7a",
    "auth": "oauth2",
    "owner": "620a3c46cf7154924cf442cb",
    "enabled": true,
    "name": "Fibery Developer",
    "masterAccountId": null,
    "lastUpdatedOn": "2022-02-21T13:30:51.350Z"
  },
  "lastSynchronizedAt": null,
  "pagination": null
}

Response example:

{
  "items": [
    {
      "id": "4455580b-000b-4313-8128-f1ca2d2dec34",
      "archived": false,
      "created_time": "2022-02-14T11:28:00.000Z",
      "last_edited_time": "2022-02-14T11:30:00.000Z",
      "__notion_link": "https://www.notion.so/Login-Page-4455580b000b43138128f1ca2d2dec34",
      "related to tasks (column)": [
        "b829daf3-bae5-40a0-a090-56a30f240a28"
      ],
      "tags": [
        "Urgent"
      ],
      "due date": "2022-02-24",
      "name": [
        "Login Page"
      ]
    },
    {
      "id": "9b3dff11-582b-498a-ba9b-571827ab3ca7",
      "archived": false,
      "created_time": "2022-02-14T11:28:00.000Z",
      "last_edited_time": "2022-02-14T11:29:00.000Z",
      "__notion_link": "https://www.notion.so/Home-Page-9b3dff11582b498aba9b571827ab3ca7",
      "related to tasks (column)": [
        "987a714b-0b7e-4b03-bdaf-c0efc5d522fb",
        "539a4d0e-6871-434b-a5cb-619f5bd5a911"
      ],
      "tags": [
        "Important",
        "Urgent"
      ],
      "due date": "2022-02-14",
      "name": [
        "Home Page"
      ]
    }
  ],
  "pagination": {
    "hasNext": false,
    "nextPageConfig": {
      "start_cursor": null,
      "schema": null
    }
  }
}

Source Code

The source code of Notion integration can be found in our public repository as well as other examples. Notion app is used in production and can be tried by following integrate link in your database editor.

Getting started

General

HTTP API

AI & agents

Integrations

GraphQL API

Tutorial: Notion sync

App Configuration

OAuth2

Token Authentication

Token Authorization

OAuth 2

Synchronizer configuration

Schema of synchronization

Data route

Source Code

​App Configuration

​OAuth2

​Token Authentication

​Token Authorization

​OAuth 2

​Synchronizer configuration

​Schema of synchronization

​Data route

​Source Code

App Configuration

OAuth2

Token Authentication

Token Authorization

OAuth 2

Synchronizer configuration

Schema of synchronization

Data route

Source Code