Create Evaluation Job

bedrock_create_evaluation_job

R Documentation

Creates an evaluation job

Description

Creates an evaluation job.

Usage

bedrock_create_evaluation_job(jobName, jobDescription,
  clientRequestToken, roleArn, customerEncryptionKeyId, jobTags,
  applicationType, evaluationConfig, inferenceConfig, outputDataConfig)

Arguments

jobName

[required] A name for the evaluation job. Names must unique with your Amazon Web Services account, and your account's Amazon Web Services region.

jobDescription

A description of the evaluation job.

clientRequestToken

A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If this token matches a previous request, Amazon Bedrock ignores the request, but does not return an error. For more information, see Ensuring idempotency.

roleArn

[required] The Amazon Resource Name (ARN) of an IAM service role that Amazon Bedrock can assume to perform tasks on your behalf. To learn more about the required permissions, see Required permissions for model evaluations.

customerEncryptionKeyId

Specify your customer managed encryption key Amazon Resource Name (ARN) that will be used to encrypt your evaluation job.

jobTags

Tags to attach to the model evaluation job.

applicationType

Specifies whether the evaluation job is for evaluating a model or evaluating a knowledge base (retrieval and response generation).

evaluationConfig

[required] Contains the configuration details of either an automated or human-based evaluation job.

inferenceConfig

[required] Contains the configuration details of the inference model for the evaluation job.

For model evaluation jobs, automated jobs support a single model or inference profile, and jobs that use human workers support two models or inference profiles.

outputDataConfig

[required] Contains the configuration details of the Amazon S3 bucket for storing the results of the evaluation job.

Value

A list with the following syntax:

list(
  jobArn = "string"
)

Request syntax

svc$create_evaluation_job(
  jobName = "string",
  jobDescription = "string",
  clientRequestToken = "string",
  roleArn = "string",
  customerEncryptionKeyId = "string",
  jobTags = list(
    list(
      key = "string",
      value = "string"
    )
  ),
  applicationType = "ModelEvaluation"|"RagEvaluation",
  evaluationConfig = list(
    automated = list(
      datasetMetricConfigs = list(
        list(
          taskType = "Summarization"|"Classification"|"QuestionAndAnswer"|"Generation"|"Custom",
          dataset = list(
            name = "string",
            datasetLocation = list(
              s3Uri = "string"
            )
          ),
          metricNames = list(
            "string"
          )
        )
      ),
      evaluatorModelConfig = list(
        bedrockEvaluatorModels = list(
          list(
            modelIdentifier = "string"
          )
        )
      )
    ),
    human = list(
      humanWorkflowConfig = list(
        flowDefinitionArn = "string",
        instructions = "string"
      ),
      customMetrics = list(
        list(
          name = "string",
          description = "string",
          ratingMethod = "string"
        )
      ),
      datasetMetricConfigs = list(
        list(
          taskType = "Summarization"|"Classification"|"QuestionAndAnswer"|"Generation"|"Custom",
          dataset = list(
            name = "string",
            datasetLocation = list(
              s3Uri = "string"
            )
          ),
          metricNames = list(
            "string"
          )
        )
      )
    )
  ),
  inferenceConfig = list(
    models = list(
      list(
        bedrockModel = list(
          modelIdentifier = "string",
          inferenceParams = "string",
          performanceConfig = list(
            latency = "standard"|"optimized"
          )
        )
      )
    ),
    ragConfigs = list(
      list(
        knowledgeBaseConfig = list(
          retrieveConfig = list(
            knowledgeBaseId = "string",
            knowledgeBaseRetrievalConfiguration = list(
              vectorSearchConfiguration = list(
                numberOfResults = 123,
                overrideSearchType = "HYBRID"|"SEMANTIC",
                filter = list(
                  equals = list(
                    key = "string",
                    value = list()
                  ),
                  notEquals = list(
                    key = "string",
                    value = list()
                  ),
                  greaterThan = list(
                    key = "string",
                    value = list()
                  ),
                  greaterThanOrEquals = list(
                    key = "string",
                    value = list()
                  ),
                  lessThan = list(
                    key = "string",
                    value = list()
                  ),
                  lessThanOrEquals = list(
                    key = "string",
                    value = list()
                  ),
                  in = list(
                    key = "string",
                    value = list()
                  ),
                  notIn = list(
                    key = "string",
                    value = list()
                  ),
                  startsWith = list(
                    key = "string",
                    value = list()
                  ),
                  listContains = list(
                    key = "string",
                    value = list()
                  ),
                  stringContains = list(
                    key = "string",
                    value = list()
                  ),
                  andAll = list(
                    list()
                  ),
                  orAll = list(
                    list()
                  )
                )
              )
            )
          ),
          retrieveAndGenerateConfig = list(
            type = "KNOWLEDGE_BASE"|"EXTERNAL_SOURCES",
            knowledgeBaseConfiguration = list(
              knowledgeBaseId = "string",
              modelArn = "string",
              retrievalConfiguration = list(
                vectorSearchConfiguration = list(
                  numberOfResults = 123,
                  overrideSearchType = "HYBRID"|"SEMANTIC",
                  filter = list(
                    equals = list(
                      key = "string",
                      value = list()
                    ),
                    notEquals = list(
                      key = "string",
                      value = list()
                    ),
                    greaterThan = list(
                      key = "string",
                      value = list()
                    ),
                    greaterThanOrEquals = list(
                      key = "string",
                      value = list()
                    ),
                    lessThan = list(
                      key = "string",
                      value = list()
                    ),
                    lessThanOrEquals = list(
                      key = "string",
                      value = list()
                    ),
                    in = list(
                      key = "string",
                      value = list()
                    ),
                    notIn = list(
                      key = "string",
                      value = list()
                    ),
                    startsWith = list(
                      key = "string",
                      value = list()
                    ),
                    listContains = list(
                      key = "string",
                      value = list()
                    ),
                    stringContains = list(
                      key = "string",
                      value = list()
                    ),
                    andAll = list(
                      list()
                    ),
                    orAll = list(
                      list()
                    )
                  )
                )
              ),
              generationConfiguration = list(
                promptTemplate = list(
                  textPromptTemplate = "string"
                ),
                guardrailConfiguration = list(
                  guardrailId = "string",
                  guardrailVersion = "string"
                ),
                kbInferenceConfig = list(
                  textInferenceConfig = list(
                    temperature = 123.0,
                    topP = 123.0,
                    maxTokens = 123,
                    stopSequences = list(
                      "string"
                    )
                  )
                ),
                additionalModelRequestFields = list(
                  list()
                )
              ),
              orchestrationConfiguration = list(
                queryTransformationConfiguration = list(
                  type = "QUERY_DECOMPOSITION"
                )
              )
            ),
            externalSourcesConfiguration = list(
              modelArn = "string",
              sources = list(
                list(
                  sourceType = "S3"|"BYTE_CONTENT",
                  s3Location = list(
                    uri = "string"
                  ),
                  byteContent = list(
                    identifier = "string",
                    contentType = "string",
                    data = raw
                  )
                )
              ),
              generationConfiguration = list(
                promptTemplate = list(
                  textPromptTemplate = "string"
                ),
                guardrailConfiguration = list(
                  guardrailId = "string",
                  guardrailVersion = "string"
                ),
                kbInferenceConfig = list(
                  textInferenceConfig = list(
                    temperature = 123.0,
                    topP = 123.0,
                    maxTokens = 123,
                    stopSequences = list(
                      "string"
                    )
                  )
                ),
                additionalModelRequestFields = list(
                  list()
                )
              )
            )
          )
        )
      )
    )
  ),
  outputDataConfig = list(
    s3Uri = "string"
  )
)