Merge pull request #35268 from github/repo-sync

Repo sync
This commit is contained in:
docs-bot 2024-11-12 19:39:54 -05:00 коммит произвёл GitHub
Родитель 9bb1b989ee f468a8dea9
Коммит ebc8601bd3
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: B5690EEEBB952194
37 изменённых файлов: 293 добавлений и 227 удалений

Просмотреть файл

@ -133,9 +133,9 @@ To prevent confusion from your developers, you can change this behavior so that
> [!NOTE]
> If a user is signed in to their personal account when they attempt to access any of your enterprise's resources, they'll be automatically signed out and redirected to SSO to sign in to their {% data variables.enterprise.prodname_managed_user %}. For more information, see "[AUTOTITLE](/enterprise-cloud@latest/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-your-personal-account/managing-multiple-accounts)."
{% data reusables.enterprise-accounts.access-enterprise %}
{% data reusables.enterprise-accounts.settings-tab %}
{% data reusables.enterprise-accounts.security-tab %}
{% data reusables.enterprise-accounts.access-enterprise-emu %}
{% data reusables.enterprise-accounts.identity-provider-tab %}
{% data reusables.enterprise-accounts.sso-configuration %}
1. Under "Single sign-on settings", select or deselect **Automatically redirect users to sign in**.
{% endif %}

Просмотреть файл

@ -47,9 +47,9 @@ OIDC does not support IdP-initiated authentication.
1. Sign into {% data variables.product.prodname_dotcom %} as the setup user for your new enterprise with the username **@SHORT-CODE_admin**.
{% data reusables.enterprise-accounts.access-enterprise-emu %}
{% data reusables.enterprise-accounts.settings-tab %}
{% data reusables.enterprise-accounts.security-tab %}
1. Under "OpenID Connect single sign-on", select **Require OIDC single sign-on**.
{% data reusables.enterprise-accounts.identity-provider-tab %}
{% data reusables.enterprise-accounts.sso-configuration %}
1. Under "OIDC single sign-on", select **Enable OIDC configuration**.
1. To continue setup and be redirected to Entra ID, click **Save**.
{% data reusables.enterprise-accounts.emu-azure-admin-consent %}
{% data reusables.enterprise-accounts.download-recovery-codes %}

Просмотреть файл

@ -104,16 +104,16 @@ After the initial configuration of SAML SSO, the only setting you can update on
> {% data reusables.enterprise-accounts.emu-password-reset-session %}
{% data reusables.enterprise-accounts.access-enterprise-emu %}
{% data reusables.enterprise-accounts.settings-tab %}
{% data reusables.enterprise-accounts.security-tab %}
{% data reusables.enterprise-accounts.identity-provider-tab %}
{% data reusables.enterprise-accounts.sso-configuration %}
1. Under "SAML single sign-on", select **Require SAML authentication**.
1. Under "SAML single sign-on", select **Add SAML configuration**.
1. Under **Sign on URL**, type the HTTPS endpoint of your IdP for SSO requests that you noted while configuring your IdP.
1. Under **Issuer**, type your SAML issuer URL that you noted while configuring your IdP, to verify the authenticity of sent messages.
1. Under **Public Certificate**, paste the certificate that you noted while configuring your IdP, to verify SAML responses.
{% data reusables.saml.edit-signature-and-digest-methods %}
1. Under **Public Certificate**, select the **Signature Method** and **Digest Method** dropdown menus, then click the hashing algorithm used by your SAML issuer.
1. Before enabling SAML SSO for your enterprise, to ensure that the information you've entered is correct, click **Test SAML configuration**. {% data reusables.saml.test-must-succeed %}
1. Click **Save**.
1. Click **Save SAML settings**.
> [!NOTE]
> After you require SAML SSO for your enterprise and save SAML settings, the setup user will continue to have access to the enterprise and will remain signed in to GitHub along with the {% data variables.enterprise.prodname_managed_users %} provisioned by your IdP who will also have access to the enterprise.

Просмотреть файл

@ -41,7 +41,7 @@ If you want to migrate to a new identity provider (IdP) or tenant rather than di
{% data reusables.emus.sign-in-as-setup-user %}
1. Attempt to access your enterprise account, and use a recovery code to bypass SAML SSO or OIDC. For more information, see "[AUTOTITLE](/admin/identity-and-access-management/managing-recovery-codes-for-your-enterprise/accessing-your-enterprise-account-if-your-identity-provider-is-unavailable)."
{% data reusables.enterprise-accounts.access-enterprise-emu %}
{% data reusables.enterprise-accounts.settings-tab %}
{% data reusables.enterprise-accounts.security-tab %}
1. Under "SAML single sign-on", deselect **Require SAML authentication** or **Require OIDC single sign-on**.
1. Click **Save**.
{% data reusables.enterprise-accounts.identity-provider-tab %}
{% data reusables.enterprise-accounts.sso-configuration %}
1. Next to "SAML single sign-on" or "OIDC single sign-on", click to deselect **SAML single sign-on** or **OIDC single sign-on**.
1. To confirm, click **Disable SAML single sign-on** or **Disable OIDC single sign-on**.

Просмотреть файл

@ -20,16 +20,23 @@ In the event that your IdP is unavailable, you can use a recovery code to sign i
If you did not save your recovery codes when you configured SSO, you can still access the codes from your enterprise's settings.
{% data reusables.enterprise-accounts.access-enterprise %}
## Downloading codes for an enterprise with personal accounts
{% data reusables.enterprise-accounts.access-enterprise-personal-accounts %}
{% data reusables.enterprise-accounts.settings-tab %}
{% data reusables.enterprise-accounts.security-tab %}
1. Under{% ifversion oidc-for-emu %} either{% endif %} "Require SAML authentication"{% ifversion oidc-for-emu %} or "Require OIDC authentication"{% endif %}, click **Save your recovery codes**.{% ifversion oidc-for-emu %}
> [!NOTE]
> OIDC SSO is only available for {% data variables.product.prodname_emus %}. For more information, see "[AUTOTITLE](/admin/identity-and-access-management/using-enterprise-managed-users-for-iam/about-enterprise-managed-users)."
{% endif %}
1. Under "Require SAML authentication", click **Save your recovery codes**.
![Screenshot of the "Authentication security" screen. The "Save your recovery codes" hyperlink is highlighted with an orange outline.](/assets/images/help/enterprises/saml-recovery-codes-link.png)
1. To save your recovery codes, click **Download**, **Print**, or **Copy**.
## Downloading codes for an enterprise with {% data variables.product.prodname_emus %}
{% data reusables.enterprise-accounts.access-enterprise-emu %}
{% data reusables.enterprise-accounts.identity-provider-tab %}
{% data reusables.enterprise-accounts.sso-configuration %}
1. Under either "SAML single sign-on" or "OIDC single sign-on", click **Save your recovery codes**.
1. To save your recovery codes, click **Download**, **Print**, or **Copy**.

Просмотреть файл

@ -181,10 +181,10 @@ If you don't use a partner IdP, or if you only use a partner IdP for authenticat
> {% data reusables.enterprise-accounts.emu-password-reset-session %}
{% data reusables.enterprise-accounts.access-enterprise-emu %}
{% data reusables.enterprise-accounts.settings-tab %}
{% data reusables.enterprise-accounts.security-tab %}
{% data reusables.enterprise-accounts.identity-provider-tab %}
{% data reusables.enterprise-accounts.sso-configuration %}
1. Under "Open SCIM Configuration", select "Enable open SCIM configuration".
1. Manage the lifecycle of your users by making calls to the REST API endpoints for SCIM provisioning. For more information, see "[AUTOTITLE](/admin/identity-and-access-management/provisioning-user-accounts-for-enterprise-managed-users/provisioning-users-and-groups-with-scim-using-the-rest-api)."
1. Manage the lifecycle of your users by making calls to the REST API endpoints for SCIM provisioning. See "[AUTOTITLE](/admin/identity-and-access-management/provisioning-user-accounts-for-enterprise-managed-users/provisioning-users-and-groups-with-scim-using-the-rest-api)."
{% endif %}

Просмотреть файл

@ -88,9 +88,8 @@ Enterprise owners can review a list of IdP groups, each group's memberships, and
{% data reusables.enterprise-accounts.access-enterprise %}
{% data reusables.enterprise-accounts.click-identity-provider %}
1. To see the members and teams connected to an IdP group, click the group's name.
1. {% data reusables.enterprise-accounts.groups-tab %}
1. To view the teams connected to the IdP group, click **Teams**.
If a team cannot sync with the group on your IdP, the team will display an error. For more information, see "[AUTOTITLE](/admin/identity-and-access-management/using-enterprise-managed-users-for-iam/troubleshooting-team-membership-with-identity-provider-groups)."

Просмотреть файл

@ -29,13 +29,13 @@ If {% data variables.product.prodname_dotcom %} is unable to synchronize team me
## Viewing errors for team synchronization with an IdP group
{% data reusables.enterprise-accounts.access-enterprise %}
1. In the list of enterprises, click the enterprise you want to view.
{% data reusables.enterprise-accounts.click-identity-provider %}
1. Under **Identity provider**, click **Groups**.
1. If synchronization for a group is experiencing problems, you'll see a message that reads "Some groups are failing to synchronize to teams. Check that you have available licenses."
1. In the list of IdP groups, click the group you'd like to review.
1. To review the synchronization error for the group, under the name of the group, click **Teams**.
![Screenshot of the page for a synchronized IdP group. Under the name of the group, to the right, the "Teams" tab is highlighted in dark orange.](/assets/images/help/enterprises/idp-group-sync-teams-tab.png)
If a team is unable to sync membership with a group on your IdP, you'll see a description of the problem under the team's name and membership count.
{% ifversion ghec %}

Просмотреть файл

@ -37,10 +37,10 @@ If you're new to {% data variables.product.prodname_emus %} and haven't yet conf
## Migrating your enterprise
{% data reusables.emus.sign-in-as-setup-user %}
{% data reusables.enterprise-accounts.access-enterprise %}
{% data reusables.enterprise-accounts.settings-tab %}
{% data reusables.enterprise-accounts.access-enterprise-emu %}
{% data reusables.emus.use-enterprise-recovery-code %}
{% data reusables.enterprise-accounts.security-tab %}
1. Deselect **Require OIDC single sign-on**.
1. Click **Save**.
1. Configure SAML authentication and SCIM provisioning. For more information, see [Tutorial: Microsoft Entra single sign-on (SSO) integration with GitHub Enterprise Managed User](https://learn.microsoft.com/entra/identity/saas-apps/github-enterprise-managed-user-tutorial) on Microsoft Learn.
{% data reusables.enterprise-accounts.identity-provider-tab %}
{% data reusables.enterprise-accounts.sso-configuration %}
1. Deselect **OIDC single sign-on**.
1. Confirm and click **Disable OIDC single sign-on**.
1. Configure SAML authentication and SCIM provisioning. See [Tutorial: Microsoft Entra single sign-on (SSO) integration with GitHub Enterprise Managed User](https://learn.microsoft.com/entra/identity/saas-apps/github-enterprise-managed-user-tutorial) on Microsoft Learn.

Просмотреть файл

@ -44,14 +44,15 @@ To migrate your enterprise from SAML to OIDC, you will disable your existing {%
> Migration of your enterprise from SAML to OIDC can take up to an hour. During the migration, users cannot access your enterprise on {% data variables.product.github %}.
1. Before you begin the migration, sign in to Azure and disable provisioning in the existing {% data variables.product.prodname_emu_idp_application %} application.
1. If you use [Conditional Access (CA) network location policies](https://docs.microsoft.com/en-us/azure/active-directory/conditional-access/location-condition) in Entra ID, and you're currently using an IP allow list with your enterprise account or any of the organizations owned by the enterprise account, disable the IP allow lists. For more information, see "[AUTOTITLE](/admin/policies/enforcing-policies-for-your-enterprise/enforcing-policies-for-security-settings-in-your-enterprise#managing-allowed-ip-addresses-for-organizations-in-your-enterprise)" and "[AUTOTITLE](/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/managing-allowed-ip-addresses-for-your-organization)."
1. If you use [Conditional Access (CA) network location policies](https://docs.microsoft.com/en-us/azure/active-directory/conditional-access/location-condition) in Entra ID, and you're currently using an IP allow list with your enterprise account or any of the organizations owned by the enterprise account, disable the IP allow lists. See "[AUTOTITLE](/admin/policies/enforcing-policies-for-your-enterprise/enforcing-policies-for-security-settings-in-your-enterprise#managing-allowed-ip-addresses-for-organizations-in-your-enterprise)" and "[AUTOTITLE](/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/managing-allowed-ip-addresses-for-your-organization)."
{% data reusables.emus.sign-in-as-setup-user %}
{% data reusables.enterprise-accounts.access-enterprise %}
{% data reusables.enterprise-accounts.settings-tab %}
{% data reusables.enterprise-accounts.access-enterprise-emu %}
{% data reusables.emus.use-enterprise-recovery-code %}
{% data reusables.enterprise-accounts.security-tab %}
1. At the bottom of the page, next to "Migrate to OpenID Connect single sign-on", click **Configure with Azure**.
1. Read the warning, then click **I understand, begin migrating to OpenID Connect**.
{% data reusables.enterprise-accounts.identity-provider-tab %}
{% data reusables.enterprise-accounts.sso-configuration %}
1. At the bottom of the page, click **Migrate to OpenID Connect single sign-on**.
1. Read the warning, then click **Migrate to OIDC**.
1. Click **Begin OIDC migration**.
{% data reusables.enterprise-accounts.emu-azure-admin-consent %}
1. After you grant consent, a new browser window will open to {% data variables.product.github %} and display a new set of recovery codes for your {% data variables.enterprise.prodname_emu_enterprise %}. Download the codes, then click **Enable OIDC authentication**.
1. Wait for the migration to complete, which can take up to an hour. To check the status of the migration, navigate to your enterprise's authentication security settings page. If "Require SAML authentication" is selected, the migration is still in progress.
@ -60,10 +61,9 @@ To migrate your enterprise from SAML to OIDC, you will disable your existing {%
> Do not provision new users from the application on Entra ID during the migration.
1. In a new tab or window, while signed in as the setup user, create a {% data variables.product.pat_v1 %} with the **scim:enterprise** scope and **no expiration** and copy it to your clipboard. For more information about creating a new token, see "[AUTOTITLE](/admin/identity-and-access-management/using-enterprise-managed-users-for-iam/configuring-scim-provisioning-for-enterprise-managed-users#creating-a-personal-access-token)."
1. In the provisioning settings for the {% data variables.product.prodname_emu_idp_oidc_application %} application in the Microsoft Entra admin center, under "Tenant URL", type the tenant URL for your enterprise:
* For **{% data variables.product.prodname_dotcom_the_website %}**: `https://api.github.com/scim/v2/enterprises/YOUR_ENTERPRISE`, replacing YOUR_ENTERPRISE with the name of your enterprise account. For example, if your enterprise account's URL is `https://github.com/enterprises/octo-corp`, the name of the enterprise account is `octo-corp`.
* For **{% data variables.enterprise.data_residency_site %}**: `https://api.SUBDOMAIN.ghe.com/scim/v2/enterprises/SUBDOMAIN`, where SUBDOMAIN is your enterprise's subdomain on {% data variables.enterprise.data_residency_site %}.
1. In the provisioning settings for the {% data variables.product.prodname_emu_idp_oidc_application %} application in the Microsoft Entra admin center, under "Tenant URL", the tenant URL for your enterprise:
* For **{% data variables.product.prodname_dotcom_the_website %}**: `https://api.github.com/scim/v2/enterprises/YOUR_ENTERPRISE`, replacing YOUR_ENTERPRISE with the name of your enterprise account. For example, if your enterprise account's URL is `https://github.com/enterprises/octo-corp`, the name of the enterprise account is `octo-corp`.
* For **{% data variables.enterprise.data_residency_site %}**: `https://api.SUBDOMAIN.ghe.com/scim/v2/enterprises/SUBDOMAIN`, where SUBDOMAIN is your enterprise's subdomain on {% data variables.enterprise.data_residency_site %}.
1. Under "Secret token", paste the {% data variables.product.pat_v1 %} with the **scim:enterprise** scope that you created earlier.
1. To test the configuration, click **Test Connection**.

Просмотреть файл

@ -0,0 +1,2 @@
1. In the top-right corner of {% data variables.product.prodname_dotcom %}, click your profile photo.
1. Click **Your enterprises**, then click the enterprise you want to view.

Просмотреть файл

@ -0,0 +1 @@
1. Under **Identity provider**, click **Groups**.

Просмотреть файл

@ -0,0 +1 @@
1. On the left side of the page, in the enterprise account sidebar, click **Identity provider**.

Просмотреть файл

@ -0,0 +1 @@
1. Under **Identity Provider**, click **Single sign-on configuration**.

Просмотреть файл

@ -5,7 +5,8 @@ import { ThumbsdownIcon, ThumbsupIcon } from '@primer/octicons-react'
import { useTranslation } from 'src/languages/components/useTranslation'
import { Link } from 'src/frame/components/Link'
import { sendEvent, EventType } from 'src/events/components/events'
import { sendEvent } from 'src/events/components/events'
import { EventType } from '../types'
import styles from './Survey.module.scss'

Просмотреть файл

@ -3,6 +3,7 @@ import Cookies from 'src/frame/components/lib/cookies'
import { parseUserAgent } from './user-agent'
import { Router } from 'next/router'
import { isLoggedIn } from 'src/frame/components/hooks/useHasAccount'
import { EventType, EventPropsByType } from '../types'
const COOKIE_NAME = '_docs-events'
@ -54,76 +55,6 @@ export function getUserEventsId() {
return cookieValue
}
export enum EventType {
page = 'page',
exit = 'exit',
link = 'link',
hover = 'hover',
search = 'search',
searchResult = 'searchResult',
survey = 'survey',
experiment = 'experiment',
preference = 'preference',
clipboard = 'clipboard',
print = 'print',
}
type SendEventProps = {
[EventType.clipboard]: {
clipboard_operation: string
clipboard_target?: string
}
[EventType.exit]: {
exit_render_duration?: number
exit_first_paint?: number
exit_dom_interactive?: number
exit_dom_complete?: number
exit_visit_duration?: number
exit_scroll_length?: number
exit_scroll_flip?: number
}
[EventType.experiment]: {
experiment_name: string
experiment_variation: string
experiment_success?: boolean
}
[EventType.hover]: {
hover_url: string
hover_samesite?: boolean
}
[EventType.link]: {
link_url: string
link_samesite?: boolean
link_samepage?: boolean
link_container?: string
}
[EventType.page]: {}
[EventType.preference]: {
preference_name: string
preference_value: string
}
[EventType.print]: {}
[EventType.search]: {
search_query: string
search_context?: string
}
[EventType.searchResult]: {
search_result_query: string
search_result_index: number
search_result_total: number
search_result_rank: number
search_result_url: string
}
[EventType.survey]: {
survey_token?: string // Honeypot, doesn't exist in schema
survey_vote: boolean
survey_comment?: string
survey_email?: string
survey_rating?: number
survey_comment_language?: string
}
}
function getMetaContent(name: string) {
const metaTag = document.querySelector(`meta[name="${name}"]`) as HTMLMetaElement
return metaTag?.content
@ -136,7 +67,7 @@ export function sendEvent<T extends EventType>({
}: {
type: T
version?: string
} & SendEventProps[T]) {
} & EventPropsByType[T]) {
const body = {
type,

Просмотреть файл

@ -1,5 +1,6 @@
import murmur from 'imurmurhash'
import { getUserEventsId, sendEvent, EventType } from './events'
import { getUserEventsId, sendEvent } from './events'
import { EventType } from '../types'
let initialized = false

Просмотреть файл

@ -13,71 +13,71 @@ export const SIGNAL_RATINGS = [
{
reduction: 1.0,
name: 'email-only',
validator: (comment) => isEmailOnly(comment),
validator: (comment: string) => isEmailOnly(comment),
},
{
reduction: 0.2,
name: 'contains-email',
validator: (comment) => isContainingEmail(comment),
validator: (comment: string) => isContainingEmail(comment),
},
{
reduction: 1.0,
name: 'url-only',
validator: (comment) => isURL(comment),
validator: (comment: string) => isURL(comment),
},
{
reduction: 1.0,
name: 'numbers-only',
validator: (comment) => isNumbersOnly(comment),
validator: (comment: string) => isNumbersOnly(comment),
},
{
reduction: 0.1,
name: 'all-uppercase',
validator: (comment) => isAllUppercase(comment),
validator: (comment: string) => isAllUppercase(comment),
},
{
reduction: 0.5,
name: 'single-word',
validator: (comment) => isSingleWord(comment),
validator: (comment: string) => isSingleWord(comment),
},
{
reduction: 0.2,
name: 'too-short',
validator: (comment) => isTooShort(comment),
validator: (comment: string) => isTooShort(comment),
},
{
reduction: 0.2,
name: 'not-language',
validator: (comment, language) => isNotLanguage(comment, language),
validator: (comment: string, language: string) => isNotLanguage(comment, language),
},
{
reduction: 0.3,
name: 'cuss-words-likely',
validator: (comment, language) => isLikelyCussWords(comment, language),
validator: (comment: string, language: string) => isLikelyCussWords(comment, language),
},
{
reduction: 0.1,
name: 'cuss-words-maybe',
validator: (comment, language) => isMaybeCussWords(comment, language),
validator: (comment: string, language: string) => isMaybeCussWords(comment, language),
},
{
reduction: 0.2,
name: 'mostly-emoji',
validator: (comment) => isMostlyEmoji(comment),
validator: (comment: string) => isMostlyEmoji(comment),
},
{
reduction: 1.0,
name: 'spammy-words',
validator: (comment) => isSpammyWordList(comment),
validator: (comment: string) => isSpammyWordList(comment),
},
]
export async function getGuessedLanguage(comment) {
export async function getGuessedLanguage(comment: string) {
if (!comment || !comment.trim()) {
return
}
const bestGuess = language.guessBest(comment.trim())
const bestGuess = language.guessBest(comment.trim(), [])
if (!bestGuess) return // Can happen if the text is just whitespace
// // @horizon-rs/language-guesser is based on tri-grams and can lead
// // to false positives. For example, it thinks that 'Thamk you ❤️🙏' is
@ -88,10 +88,10 @@ export async function getGuessedLanguage(comment) {
// // But are they useful comments? Given that this is just a signal,
// // and not a hard blocker, it's more of a clue than a fact.
return bestGuess.alpha2
return bestGuess.alpha2 || undefined
}
export async function analyzeComment(text, language = 'en') {
export async function analyzeComment(text: string, language = 'en') {
const signals = []
let rating = 1.0
for (const { reduction, name, validator } of SIGNAL_RATINGS) {
@ -105,7 +105,7 @@ export async function analyzeComment(text, language = 'en') {
return { signals, rating }
}
function isEmailOnly(text) {
function isEmailOnly(text: string) {
if (text.includes('@') && !/\s/.test(text.trim()) && !text.includes('://')) {
const atSigns = text.split('@').length
if (atSigns === 2) {
@ -114,7 +114,7 @@ function isEmailOnly(text) {
}
}
function isContainingEmail(text) {
function isContainingEmail(text: string) {
if (text.includes('@') && !isEmailOnly(text)) {
// Don't use splitWords() here because `foo@example.com` will be
// split up into ['foo', 'example.com'].
@ -123,35 +123,35 @@ function isContainingEmail(text) {
return false
}
function isURL(text) {
function isURL(text: string) {
if (!text.trim().includes(' ')) {
if (URL.canParse(text.trim())) return true
}
}
function isNumbersOnly(text) {
function isNumbersOnly(text: string) {
return /^\d+$/.test(text.replace(/\s/g, ''))
}
function isAllUppercase(text) {
function isAllUppercase(text: string) {
return /[A-Z]/.test(text) && text === text.toUpperCase()
}
function isTooShort(text) {
function isTooShort(text: string) {
const split = text.trim().split(/\s+/)
if (split.length <= 3) {
return true
}
}
function isSingleWord(text) {
function isSingleWord(text: string) {
const whitespaceSplit = text.trim().split(/\s+/)
// E.g. `this-has-no-whitespace` or `snap/hooks/install`
return whitespaceSplit.length === 1
}
function isNotLanguage(text, language_) {
const bestGuess = language.guessBest(text.trim())
function isNotLanguage(text: string, language_: string) {
const bestGuess = language.guessBest(text.trim(), [])
if (!bestGuess) return true // Can happen if the text is just whitespace
// @horizon-rs/language-guesser is based on tri-grams and can lead
// to false positives. For example, it thinks that 'Thamk you ❤️🙏' is
@ -167,7 +167,7 @@ function isNotLanguage(text, language_) {
return bestGuess.alpha2 !== language_ && bestGuess.alpha2 !== 'en'
}
function isMostlyEmoji(text) {
function isMostlyEmoji(text: string) {
text = text.replace(/\s/g, '')
const emojiRegex = /\p{Emoji}/gu
const emojiMatches = text.match(emojiRegex)
@ -176,7 +176,7 @@ function isMostlyEmoji(text) {
return emojiRatio > 0.25
}
function getCussWords(lang) {
function getCussWords(lang: string) {
switch (lang) {
case 'pt':
return cussPt
@ -189,9 +189,9 @@ function getCussWords(lang) {
}
}
function isLikelyCussWords(text, language_, rating = 2) {
function isLikelyCussWords(text: string, language_: string, rating = 2) {
const cussWords = getCussWords(language_)
const words = splitWords(text, language_ || 'en').map((word) => word.toLowerCase())
const words = splitWords(text).map((word) => word.toLowerCase())
for (const word of words) {
if (cussWords[word] && cussWords[word] === rating) {
return true
@ -200,21 +200,23 @@ function isLikelyCussWords(text, language_, rating = 2) {
return false
}
function isMaybeCussWords(text, language_) {
function isMaybeCussWords(text: string, language_: string) {
return isLikelyCussWords(text, language_, 1)
}
const segmenter = new Intl.Segmenter([], { granularity: 'word' })
function splitWords(text) {
function splitWords(text: string) {
const segmentedText = segmenter.segment(text)
return [...segmentedText].filter((s) => s.isWordLike).map((s) => s.segment)
}
const surveyYaml = yaml.load(fs.readFileSync('data/survey-words.yml', 'utf8'))
const surveyWords = surveyYaml.words.map((word) => word.toLowerCase())
const surveyYaml = yaml.load(fs.readFileSync('data/survey-words.yml', 'utf8')) as {
words: string[]
}
const surveyWords = surveyYaml.words.map((word: string) => word.toLowerCase())
function isSpammyWordList(text) {
function isSpammyWordList(text: string) {
const words = text.toLowerCase().split(/(\s+|\\n+)/g)
// Currently, we're intentionally not checking for
// survey words that are substrings of a comment word.

1
src/events/lib/get-document-type.d.ts поставляемый Normal file
Просмотреть файл

@ -0,0 +1 @@
export function getDocumentType(relativePath: string): string

Просмотреть файл

@ -2,9 +2,9 @@ import { createHmac } from 'crypto'
import { Agent } from 'node:https'
import got from 'got'
import { isNil } from 'lodash-es'
import statsd from '#src/observability/lib/statsd.js'
import { report } from '#src/observability/lib/failbot.js'
import { MAX_REQUEST_TIMEOUT } from '#src/frame/lib/constants.js'
import statsd from 'src/observability/lib/statsd.js'
import { report } from 'src/observability/lib/failbot.js'
import { MAX_REQUEST_TIMEOUT } from 'src/frame/lib/constants.js'
const TIME_OUT_TEXT = 'ms has passed since batch creation'
const SERVER_DISCONNECT_TEXT = 'The server disconnected before a response was received'
@ -22,14 +22,13 @@ if (inProd && (isNil(HYDRO_SECRET) || isNil(HYDRO_ENDPOINT))) {
)
}
/*
`events` can be either like:
{schema, value}
or
[{schema, value}, {schema, value}, ...]
*/
type EventT = {
schema: string
value: Record<string, any>
}
async function _publish(
events,
events: EventT | EventT[],
{ secret, endpoint } = { secret: HYDRO_SECRET, endpoint: HYDRO_ENDPOINT },
) {
if (!secret || !endpoint) {

Просмотреть файл

@ -1,5 +1,6 @@
import { pick, snakeCase } from 'lodash-es'
import { randomUUID } from 'crypto'
import { ErrorObject } from 'ajv'
// https://ajv.js.org/api.html#error-objects
const errorKeys = [
@ -14,7 +15,7 @@ const errorKeys = [
'data',
]
export function formatErrors(errors, body) {
export function formatErrors(errors: ErrorObject[], body: any) {
return errors.map((error) => ({
event_id: randomUUID(),
version: '1.0.0',
@ -33,6 +34,6 @@ export function formatErrors(errors, body) {
}
// Leave strings alone, otherwise convert to either string or undefined
function makeString(value) {
function makeString(value: any) {
return typeof value === 'string' ? value : JSON.stringify(value)
}

Просмотреть файл

@ -1,7 +1,7 @@
import { languageKeys } from '#src/languages/lib/languages.js'
import { allVersionKeys } from '#src/versions/lib/all-versions.js'
import { productIds } from '#src/products/lib/all-products.js'
import { allTools } from '#src/tools/lib/all-tools.js'
import { allTools } from 'src/tools/lib/all-tools.js'
const versionPattern = '^\\d+(\\.\\d+)?(\\.\\d+)?$' // eslint-disable-line
@ -552,7 +552,7 @@ export const hydroNames = {
print: 'docs.v0.PrintEvent',
preference: 'docs.v0.PreferenceEvent',
validation: 'docs.v0.ValidationEvent',
}
} as Record<keyof typeof schemas, string>
const schemasKeys = Object.keys(schemas)
const hydroNamesKeys = Object.keys(hydroNames)

Просмотреть файл

@ -1,25 +1,31 @@
import express from 'express'
import { omit, without, mapValues } from 'lodash-es'
import QuickLRU from 'quick-lru'
import { ErrorObject } from 'ajv'
import type { ExtendedRequest } from '@/types'
import type { Response } from 'express'
import { schemas, hydroNames } from './lib/schema.js'
import catchMiddlewareError from '#src/observability/middleware/catch-middleware-error.js'
import { noCacheControl } from '#src/frame/middleware/cache-control.js'
import { getJsonValidator } from '#src/tests/lib/validate-json-schema.js'
import catchMiddlewareError from 'src/observability/middleware/catch-middleware-error'
import { noCacheControl } from 'src/frame/middleware/cache-control'
import { getJsonValidator } from 'src/tests/lib/validate-json-schema'
import { formatErrors } from './lib/middleware-errors.js'
import { publish as _publish } from './lib/hydro.js'
import { analyzeComment, getGuessedLanguage } from './analyze-comment.js'
import { analyzeComment, getGuessedLanguage } from './lib/analyze-comment.js'
import { EventType, EventProps, EventPropsByType } from './types'
const router = express.Router()
const OMIT_FIELDS = ['type']
const allowedTypes = new Set(without(Object.keys(schemas), 'validation'))
const isProd = process.env.NODE_ENV === 'production'
const validators = mapValues(schemas, (schema) => getJsonValidator(schema))
// In production, fire and not wait to respond.
// _publish will send an error to failbot,
// so we don't get alerts but we still track it.
// This ends up being the same as try > await > catch > (do nothing).
async function publish(...args) {
async function publish(...args: Parameters<typeof _publish>) {
if (isProd) {
_publish(...args)
return
@ -34,32 +40,33 @@ const sentValidationErrors = new QuickLRU({
router.post(
'/',
catchMiddlewareError(async function postEvents(req, res) {
catchMiddlewareError(async function postEvents(req: ExtendedRequest, res: Response) {
noCacheControl(res)
// Make sure the type is supported before continuing
const { type } = req.body
if (!type || !allowedTypes.has(type)) {
if (!req.body.type || !allowedTypes.has(req.body.type)) {
return res.status(400).json({ message: 'Invalid type' })
}
const type: EventType = req.body.type
const body: EventProps & EventPropsByType[EventType] = req.body
// Validate the data matches the corresponding data schema
const validate = validators[type]
if (!validate(req.body)) {
const hash = `${req.ip}:${validate.errors
.map((error) => error.message + error.instancePath)
.join(':')}`
// This protects so we don't bother sending the same validation
// error, per user, more than once (per time interval).
// This helps if we're bombarded with junk bot traffic. So it
// protects our Hydro instance from being overloaded with things
// that aren't helping anybody.
const hash = `${req.ip}:${(validate.errors || [])
.map((error: ErrorObject) => error.message + error.instancePath)
.join(':')}`
if (!sentValidationErrors.has(hash)) {
sentValidationErrors.set(hash, true)
// Track validation errors in Hydro so that we can know if
// there's a widespread problem in events.ts
await publish(
formatErrors(validate.errors, req.body).map((error) => ({
formatErrors(validate.errors || [], body).map((error) => ({
schema: hydroNames.validation,
value: error,
})),
@ -69,28 +76,36 @@ router.post(
return res.status(400).json(isProd ? {} : validate.errors)
}
if (type === 'survey' && req.body.survey_comment) {
req.body.survey_rating = await getSurveyCommentRating({
comment: req.body.survey_comment,
language: req.body.context.path_language,
if (isSurvey(body) && body.survey_comment) {
body.survey_rating = await getSurveyCommentRating({
comment: body.survey_comment,
language: body.context.path_language || 'en',
})
req.body.survey_comment_language = await getGuessedLanguage(req.body.survey_comment)
body.survey_comment_language = await getGuessedLanguage(body.survey_comment)
}
await publish({
schema: hydroNames[type],
value: omit(req.body, OMIT_FIELDS),
value: omit(body, OMIT_FIELDS),
})
return res.json({})
}),
)
async function getSurveyCommentRating({ comment, language }) {
if (!comment || !comment.trim()) {
return
}
// https://www.typescriptlang.org/docs/handbook/2/narrowing.html#using-type-predicates
function isSurvey(
body: EventProps & EventPropsByType[EventType],
): body is EventProps & EventPropsByType[EventType.survey] {
return body.type === EventType.survey
}
type GetSurveyCommentRatingArgs = {
comment: string
language: string
}
async function getSurveyCommentRating({ comment, language }: GetSurveyCommentRatingArgs) {
if (!comment || !comment.trim()) return
const { rating } = await analyzeComment(comment, language)
return rating
}

Просмотреть файл

@ -16,7 +16,7 @@ import util from 'node:util'
import chalk from 'chalk'
import { program } from 'commander'
import { SIGNAL_RATINGS } from '../analyze-comment'
import { SIGNAL_RATINGS } from '../lib/analyze-comment'
type Options = {
language?: string

Просмотреть файл

@ -13,7 +13,7 @@ import chalk from 'chalk'
import { parse } from 'csv-parse'
import { program } from 'commander'
import { SIGNAL_RATINGS } from '../analyze-comment'
import { SIGNAL_RATINGS } from '../lib/analyze-comment'
type Options = {
outputFile: string
@ -42,6 +42,9 @@ async function main(csvFile: string[], options: Options) {
type Record = {
[key: string]: string | number
} & {
survey_comment: string
survey_comment_language: string
}
async function analyzeFile(csvFile: string, options: Options) {
@ -57,9 +60,7 @@ async function analyzeFile(csvFile: string, options: Options) {
if (headers === null) {
headers = record as string[]
} else {
const obj: {
[key: string]: string
} = {}
const obj = {} as Record
for (let i = 0; i < headers.length; i++) {
obj[headers[i]] = record[i]
}

Просмотреть файл

@ -1,6 +1,6 @@
import { describe, expect, test } from 'vitest'
import { analyzeComment, getGuessedLanguage } from '../analyze-comment.js'
import { analyzeComment, getGuessedLanguage } from '../lib/analyze-comment.js'
describe('analyzeComment', () => {
test('email only', async () => {

Просмотреть файл

Просмотреть файл

@ -1,25 +0,0 @@
import { describe, expect, test } from 'vitest'
import { validateJson } from '#src/tests/lib/validate-json-schema.js'
import { formatErrors } from '../lib/middleware-errors.js'
import { schemas } from '../lib/schema.js'
describe('formatErrors', () => {
test('should produce objects that match the validation spec', () => {
expect.extend({
toMatchSchema(data, schema) {
const { isValid, errors } = validateJson(schema, data)
return {
pass: isValid,
message: () => (isValid ? '' : errors.message),
}
},
})
// Produce an error
const { errors } = validateJson({ type: 'string' }, 0)
const formattedErrors = formatErrors(errors, '')
for (const formatted of formattedErrors) {
expect(formatted).toMatchSchema(schemas.validation)
}
})
})

Просмотреть файл

@ -0,0 +1,19 @@
import { describe, test } from 'vitest'
import { validateJson } from 'src/tests/lib/validate-json-schema.js'
import { formatErrors } from '../lib/middleware-errors.js'
import { schemas } from '../lib/schema.js'
describe('formatErrors', () => {
test('should produce objects that match the validation spec', () => {
// Produce an error
const { errors } = validateJson({ type: 'string' }, 0)
const formattedErrors = formatErrors(errors || [], '')
for (const formatted of formattedErrors) {
const { isValid, errors } = validateJson(schemas.validation, formatted)
if (!isValid) {
throw new Error(errors?.map((e) => e.message).join(' -- '))
}
}
})
})

Просмотреть файл

@ -1,11 +1,11 @@
import { describe, expect, test, vi } from 'vitest'
import { post } from '#src/tests/helpers/e2etest.js'
import { post } from 'src/tests/helpers/e2etest.js'
describe('POST /events', () => {
vi.setConfig({ testTimeout: 60 * 1000 })
async function checkEvent(data) {
async function checkEvent(data: any) {
const body = JSON.stringify(data)
const res = await post('/api/events', {
body,

Просмотреть файл

@ -1,6 +1,6 @@
import { describe, expect, test } from 'vitest'
import { parseUserAgent } from '../components/user-agent.ts'
import { parseUserAgent } from '../components/user-agent'
describe('parseUserAgent', () => {
test('android, chrome', () => {

104
src/events/types.ts Normal file
Просмотреть файл

@ -0,0 +1,104 @@
export enum EventType {
page = 'page',
exit = 'exit',
link = 'link',
hover = 'hover',
search = 'search',
searchResult = 'searchResult',
survey = 'survey',
experiment = 'experiment',
preference = 'preference',
clipboard = 'clipboard',
print = 'print',
}
export type EventProps = {
type: EventType
version: string
context: {
event_id: string
user: string
version: string
created: string
page_event_id: string
referrer: string
href: string
hostname: string
path: string
search: string
hash: string
path_language: string
path_version: string
path_article: string
path_document_type: string
path_type: string
status: number
is_logged_in: boolean
os: string
os_version: string
browser: string
browser_version: string
timezone: number
user_language: string
application_preference: string
color_mode_preference: string
os_preference: string
code_display_preference: string
}
}
export type EventPropsByType = {
[EventType.clipboard]: {
clipboard_operation: string
clipboard_target?: string
}
[EventType.exit]: {
exit_render_duration?: number
exit_first_paint?: number
exit_dom_interactive?: number
exit_dom_complete?: number
exit_visit_duration?: number
exit_scroll_length?: number
exit_scroll_flip?: number
}
[EventType.experiment]: {
experiment_name: string
experiment_variation: string
experiment_success?: boolean
}
[EventType.hover]: {
hover_url: string
hover_samesite?: boolean
}
[EventType.link]: {
link_url: string
link_samesite?: boolean
link_samepage?: boolean
link_container?: string
}
[EventType.page]: {}
[EventType.preference]: {
preference_name: string
preference_value: string
}
[EventType.print]: {}
[EventType.search]: {
search_query: string
search_context?: string
}
[EventType.searchResult]: {
search_result_query: string
search_result_index: number
search_result_total: number
search_result_rank: number
search_result_url: string
}
[EventType.survey]: {
survey_token?: string // Honeypot, doesn't exist in schema
survey_vote: boolean
survey_comment?: string
survey_email?: string
survey_rating?: number
survey_comment_language?: string
}
}

Просмотреть файл

@ -1,5 +1,6 @@
import Cookies from 'src/frame/components/lib/cookies'
import { sendEvent, EventType } from 'src/events/components/events'
import { sendEvent } from 'src/events/components/events'
import { EventType } from 'src/events/types'
enum annotationMode {
Beside = 'beside',

Просмотреть файл

@ -6,7 +6,8 @@ import { useTranslation } from 'src/languages/components/useTranslation'
import { Box, Flash, FormControl, Spinner, TextInput } from '@primer/react'
import { Dialog } from '@primer/react/experimental'
import { useEditableDomainName } from './useEditableDomainContext'
import { sendEvent, EventType } from 'src/events/components/events'
import { sendEvent } from 'src/events/components/events'
import { EventType } from 'src/events/types'
type Props = {
xs?: boolean

Просмотреть файл

@ -7,7 +7,8 @@ import { useTranslation } from 'src/languages/components/useTranslation'
import { DEFAULT_VERSION, useVersion } from 'src/versions/components/useVersion'
import { useQuery } from 'src/search/components/useQuery'
import { useBreakpoint } from 'src/search/components/useBreakpoint'
import { EventType, sendEvent } from 'src/events/components/events'
import { sendEvent } from 'src/events/components/events'
import { EventType } from 'src/events/types'
type Props = { isSearchOpen: boolean }

Просмотреть файл

@ -6,7 +6,8 @@ import cx from 'classnames'
import { useTranslation } from 'src/languages/components/useTranslation'
import { Link } from 'src/frame/components/Link'
import { sendEvent, EventType } from 'src/events/components/events'
import { sendEvent } from 'src/events/components/events'
import { EventType } from 'src/events/types'
import styles from './SearchResults.module.scss'

Просмотреть файл

@ -1,7 +1,8 @@
import { useEffect, useState } from 'react'
import Cookies from 'src/frame/components/lib/cookies'
import { UnderlineNav } from '@primer/react'
import { sendEvent, EventType } from 'src/events/components/events'
import { sendEvent } from 'src/events/components/events'
import { EventType } from 'src/events/types'
import { useRouter } from 'next/router'
type Option = {