CT-AI Certified Tester AI Testing Exam Questions and Answers

When written requirements are given in text documents, the best way to generate test cases is by using Natural Language Processing (NLP). Here's why:

Natural Language Processing (NLP): NLP can analyze and understand human language. It can be used to process textual requirements to extract relevant information and generate test cases. This method is efficient in handling large volumes of textual data and identifying key elements necessary for testing.

Why Not Other Options:

Analyzing source code for generating test cases: This is more suitable for white-box testing where the code is available, but it doesn't apply to text-based requirements.

Machine learning on logs of execution: This approach is used for dynamic analysis based on system behavior during execution rather than static textual requirements.

GUI analysis by computer vision: This is used for testing graphical user interfaces and is not applicable to text-based requirements.

References:This aligns with the methodology discussed in the syllabus under the section on using AI for generating test cases from textual requirements​​.

Bias in AI models often originates fromincomplete or non-representative training data. In this case, if the training datasetpurposely excludes specific faulty conditions, the machine learning model willfail to learn and detectthese conditions in real-world scenarios.

This results in:

Sample bias, where the training data is not fully representative of all possible faulty conditions.

Algorithmic bias, where the model prioritizes certain defect types while ignoring others.

B. Selecting training data by purposely including all known faulty conditions→ This would help reduce bias by improving model generalization.

C. Selecting testing data from a different dataset than the training dataset→ This is a good practice to evaluate model generalization but does not inherently introduce bias.

D. Selecting testing data from a boat manufacturer's bolt longevity data→ While using unrelated data can createpoor model accuracy, it does not directly introduce bias unless systematic patterns in the incorrect dataset lead to unfair decision-making.

Section 8.3 - Testing for Algorithmic, Sample, and Inappropriate Biasstates thatsample bias can occur if the training dataset is not fully representative of the expected data space, leading to biased predictions​.

Why are the other options incorrect?Reference from ISTQB Certified Tester AI Testing Study Guide:

Clustering is a form ofunsupervised learning, which groups data points based onsimilarities without predefined labels. According toISTQB CT-AI Syllabus, clustering is used in scenarios where:

The objective is to find natural groupings in data.

The dataset does not have labeled outputs.

Patterns and structures need to be identified automatically​.

Analyzing the answer choices:

A. Associating shoppers with their shopping tendencies → Correct

Shoppers can be grouped based on purchasing behaviors(e.g., luxury shoppers vs. budget-conscious shoppers), which is a typical clustering application in market segmentation.

B. Grouping individual fish together based on their types of fins → Incorrect

If thetypes of fins are labeled, it becomes aclassification problem, which requires supervised learning.

C. Classifying muffin purchases based on packaging attractiveness → Incorrect

Classification, not clustering, because attractiveness scores or labels must be predefined.

D. Estimating the expected purchase of cat food after an ad campaign → Incorrect

This is a prediction task, best suited forregression models, which are part of supervised learning.

Thus,Option A is the best answer, asclusteringis used togroup shoppers based on tendencies without predefined labels​.

Certified Tester AI Testing Study Guide References:

ISTQB CT-AI Syllabus v1.0, Section 3.1.2 (Unsupervised Learning - Clustering and Association)

ISTQB CT-AI Syllabus v1.0, Section 3.3 (Selecting a Form of ML - Clustering)​.

AI-based systems have specific quality characteristics, includingevolution,autonomy, andadaptability. A test objective that evaluates whether an AI systemevolvesto improve performance over time directly aligns with AI quality characteristics​.

Explanation of Answer Choices:

Option A: The robot must evolve to optimize its routing.

Correct.Evolution is an AI quality characteristic that ensures the systemlearns from past experiencesand adapts to improve efficiency​.

Option B: The robot must recharge for no more than six hours a day.

Incorrect.This is an operational constraint rather than an AI-specific quality characteristic​.

Option C: The robot must record the time of each delivery which is compiled into a report.

Incorrect.Logging data does not relate to AI quality characteristics likeadaptability or autonomy​.

Option D: The robot must complete 99.99% of its deliveries each day.

Incorrect.This is a performance target rather than an AI quality characteristic​.

ISTQB CT-AI Syllabus References:

Evolution as an AI Quality Characteristic:"Check how well the system learns from its own experience. Check how well the system copes when the profile of data changes (i.e., concept drift)"​.

Thus,Option A is the best choice as it directly tests an AI quality characteristic (evolution) in the upgraded autonomous robot.

The question asks which characteristic is least likely to cause safety-related issues for an AI system. Let's evaluate each option:

Non-determinism (A): Non-deterministic systems can produce different outcomes even with the same inputs, which can lead to unpredictable behavior and potential safety issues.

Robustness (B): Robustness refers to the ability of the system to handle errors, anomalies, and unexpected inputs gracefully. A robust system is less likely to cause safety issues because it can maintain functionality under varied conditions.

High complexity (C): High complexity in AI systems can lead to difficulties in understanding, predicting, and managing the system's behavior, which can cause safety-related issues.

Self-learning (D): Self-learning systems adapt based on new data, which can lead to unexpected changes in behavior. If not properly monitored and controlled, this can result in safety issues.

References:

ISTQB CT-AI Syllabus Section 2.8 on Safety and AI discusses various factors affecting the safety of AI systems, emphasizing the importance of robustness in maintaining safe operation.

Bias in an AI system occurs when the training data contains inherent prejudices that cause the model to make unfair predictions. Experience-based testing, particularlyExploratory Data Analysis (EDA), helps uncover these biases by analyzing patterns, distributions, and potential discriminatory factors in the training data​.

Option A:“Experience-based testing should be used to confirm that the training data set is operationally relevant. This can include applying exploratory data analysis (EDA) to check for bias within the training data set.”

This is the correct answer. EDA involves examining the dataset for bias, inconsistencies, or missing values, ensuring fairness in ML model predictions​.

Option B:“Back-to-back testing should be used to compare the model created using the training data set to another model created using the test data set. If the two models significantly differ, it will indicate there is bias in the original model.”

Back-to-back testing is used for regression testing and to compare versions of an AI system but is not primarily used to detect bias​.

Option C:“Acceptance testing should be used to make sure the algorithm is suitable for the customer. The team can re-work the acceptance criteria such that the algorithm is sure to correctly predict the remaining applicants that have been set aside for the validation data set ensuring no bias is present.”

Acceptance testing focuses on meeting predefined business requirements rather than detecting and mitigating bias​.

Option D:“A/B testing should be used to verify that the test data set does not detect any bias that might have been introduced by the original training data. If the two models significantly differ, it will indicate there is bias in the original model.”

A/B testing is used for evaluating variations of a model rather than for explicitly identifying bias​.

Bias Testing Methods:"AI-based systems should be tested for algorithmic bias, sample bias, and inappropriate bias. Experience-based testing and EDA are useful for detecting bias"​.

Exploratory Data Analysis (EDA):"EDA helps uncover potential bias in training data through visualization and statistical analysis"​.

Analysis of the Answer Options:ISTQB CT-AI Syllabus References:Thus,Option A is the best choice for detecting bias in the loan applicant model.

The question asks which deficiency is most likely to be discovered by the test expert given the scenario of poor real-world performance despite good isolated accuracy.

A lack of similarity between the training and testing data (A): This is a common issue in ML where the model performs well on training data but poorly on real-world data due to a lack of representativeness in the training data. This leads to poor generalization to new, unseen data.

The input data has not been tested for quality prior to use for testing (B): While data quality is important, this option is less likely to be the primary reason for the described issue compared to the representativeness of training data.

A lack of focus on choosing the right functional-performance metrics (C): Proper metrics are crucial, but the issue described seems more related to the data mismatch rather than metric selection.

A lack of focus on non-functional requirements testing (D): Non-functional requirements are important, but the scenario specifically mentions issues with detecting real cancer cases, pointing more towards data issues.

References:

ISTQB CT-AI Syllabus Section 4.2 on Training, Validation, and Test Datasets emphasizes the importance of using representative datasets to ensure the model generalizes well to real-world data.

Sample Exam Questions document, Question #40 addresses issues related to data representativeness and model generalization.

A. This is an example of expert system bias.

Expert system bias refers to bias introduced by the rules or logic defined by experts in the system, not by the data distribution.

B. This is an example of sample bias.

Sample bias occurs when the training data is not representative of the overall population that the model will encounter in practice. In this case, the over-representation of ethnicity A (70%) compared to B, C, and D (30%) creates a sample bias, as the model may become biased towards better performance on ethnicity A.

C. This is an example of hyperparameter bias.

Hyperparameter bias relates to the settings and configurations used during the training process, not the data distribution itself.

D. This is an example of algorithmic bias.

Algorithmic bias refers to biases introduced by the algorithmic processes and decision-making rules, not directly by the distribution of training data.

Based on the provided information, optionB(sample bias) best describes the situation because the training data is skewed towards ethnicity A, potentially leading to biased model performance​​.

Understanding Supervised Learning - RegressionSupervised learning is a category of machine learning where the model is trained on labeled data. Within this category,regressionis used when the goal is to predict a continuous numeric value.

Regressiondeals with problems where the output variable is continuous in nature, meaning it can take any numerical value within a range.

Common examples include predicting prices, estimating demand, and analyzing production trends.

(A) Determining the optimal age for a chicken's egg-laying production using input data of the chicken's age and average daily egg production for one million chickens.✅(Correct)

This is a classicregression problembecause it involves predicting a continuous variable:daily egg productionbased on the input variablechicken's age.

The goal is to find a numerical relationship between age and egg production, which makesregression the appropriate supervised learning method.

(B) Recognizing a knife in carry-on luggage at a security checkpoint in an airport scanner.❌(Incorrect)

This is animage recognition task, which falls underclassification, not regression.

Classification problems involve assigning inputs to discrete categories (e.g., "knife detected" or "no knife detected").

(C) Determining if an animal is a pig or a cow based on image recognition.❌(Incorrect)

This is anotherclassification problemwhere the goal is to categorize an image into one of two labels (pig or cow).

(D) Predicting shopper purchasing behavior based on the category of shopper and the positioning of promotional displays within a store.❌(Incorrect)

This problem could involve a mix ofclassificationandassociation rule learning, but it does not explicitly predict a continuous variable in the way regression does.

Regression is used when predicting a numeric output."Predicting the age of a person based on input data about their habits or predicting the future prices of stocks are examples of problems that use regression."​

Supervised learning problems are divided into classification and regression."If the output is numeric and continuous in nature, it may be regression."​

Regression is commonly used for predicting numerical trends over time."Regression models result in a numerical or continuous output value for a given input."​

Analysis of Answer ChoicesReferences from ISTQB Certified Tester AI Testing Study GuideThus,option A is the correct answer, as it aligns with the principles of regression-based supervised learning.

Object recognition issues in test automation often arise whendevelopers frequently change object identifiersin the GUI. AI can enhance the stability of GUI automation by:

Using multiple criteria for object identification

AI cantrack UI elements using multiple attributessuch asXPath, label, ID, class, and screen coordinatesrather than relying on a single identifier that may change over time.

This approach makes the automationless brittle and more adaptive to changes in the UI​.

Why other options are incorrect?

B (Ignore unrecognizable objects to avoid failures): Ignoring objects instead of identifying them properly wouldlead to incomplete or incorrect test execution.

C (Dynamically name objects and alter source code): AI-based testing tools donot modify application source code; they work byadjusting the recognition strategy.

D (Anticipate developer changes and pre-alter automation code): While AI can adapt,it does not predict future changes to the GUI, making this option unrealistic​.

Thus,Option A is the best answer, as AI tools enhance object recognitionby dynamically selecting the most stable and persistent identification methods, improving test automation reliability.

Certified Tester AI Testing Study Guide References:

ISTQB CT-AI Syllabus v1.0, Section 11.6.1 (Using AI to Test Through the Graphical User Interface (GUI))

ISTQB CT-AI Syllabus v1.0, Section 11.6.2 (Using AI to Test the GUI)​.

Decision trees are a widely usedmachine learning (ML) techniquethat falls undersupervised learning. They are used for bothclassification and regressiontasks and are popular due to their interpretability and effectiveness.

How Decision Trees Work:

The model splits the dataset into branches based on feature conditions.

It continues to divide the data until each subset belongs to a single category (classification) or predicts a continuous value (regression).

The final result is a tree structure where decisions are made atnodes, and predictions are given atleaf nodes.

Common Applications of Decision Trees:

Fraud detection

Medical diagnosis

Customer segmentation

Recommendation systems

B (Equivalence Partitioning):This is asoftware testing technique, not a machine learning method. It is used to divide input data into partitions to reduce test cases while maintaining coverage​.

C (Boundary Value Analysis):Anothersoftware testing technique, used to check edge cases around input boundaries​.

D (Decision Tables):A structuredtesting techniqueused to validate business rules and logic, not a machine learning method​.

ISTQB CT-AI Syllabus (Section 3.1: Forms of Machine Learning - Decision Trees)

"Decision trees are used in classification and regression models and are fundamental ML algorithms"​.

Why Other Options Are Incorrect:Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:Sincedecision trees are a core technique in machine learning, while the other options aresoftware testing techniques, thecorrect answer is A.

 Coverage for Neuron Activation Values:Sign change coverage is used to ensure that test cases cause each neuron to achieve both positive and negative activation values. This type of coverage ensures that the neurons are thoroughly tested under different activation states.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 6.2 Coverage Measures for Neural Networks, which details different types of coverage measures, including sign change coverage​​.

A/B testing is a statistical testing method that compares two different versions of a system to determine which one performs better. In this scenario, theold NLP algorithmwas rated for accuracy, and after the update, thenew algorithmwas also rated by users. A statistical test was performed to compare the two versions, which is the fundamental approach ofA/B testing.

A/B testing is commonly used in:

User experience testing(e.g., comparing different versions of a website).

ML model evaluation(e.g., comparing two AI-based classifiers).

Performance assessment(e.g., determining if a new recommendation algorithm is more effective).

This approach allows for data-driven decisions, ensuring that any changes to the system result in meaningful improvements.

Section 9.4 - A/B Testingstates that A/B testing is used to compare updates in AI-based systems to determine if the newer version is better​.

Reference from ISTQB Certified Tester AI Testing Study Guide:

Data mislabeling occurs for several reasons, which can significantly impact the performance of machine learning (ML) models, especially in supervised learning. According to the ISTQB Certified Tester AI Testing (CT-AI) syllabus, mislabeling of data can be caused by the following factors:

Random errors by annotators– Mistakes made due to accidental misclassification.

Systemic errors– Errors introduced by incorrect labeling instructions or poor training of annotators.

Deliberate errors– Errors introduced intentionally by malicious data annotators.

Translation errors– Occur when correctly labeled data in one language is incorrectly translated into another language.

Subjectivity in labeling– Some labeling tasks require subjective judgment, leading to inconsistencies between different annotators.

Lack of domain knowledge– If annotators do not have sufficient expertise in the domain, they may label data incorrectly due to misunderstanding the context.

Complex classification tasks– The more complex the task, the higher the probability of labeling mistakes​.

Among the answer choices provided, "Lack of domain knowledge" (Option A) is the best answer because expertise is essential to accurately labeling data in complex domains such as medical, legal, or engineering fields.

Certified Tester AI Testing Study Guide References:

ISTQB CT-AI Syllabus v1.0, Section 4.5.2 (Mislabeled Data in Datasets)

ISTQB CT-AI Syllabus v1.0, Section 4.3 (Dataset Quality Issues)

 A. Using a pixel comparison of the GUI before and after the change to check the differences.

Pixel comparison is a traditional method and does not involve AI. It compares images at the pixel level, which can be effective but is not an intelligent approach. It is not considered an AI usage and is the least likely usage of AI for detecting GUI changes.

 B. Using computer vision to compare the GUI before and after the test object changes.

Computer vision involves using AI techniques to interpret and process images. It is a likely usage of AI for detecting changes in the GUI.

 C. Using vision-based detection of the GUI layout changes before and after test object changes.

Vision-based detection is another AI technique where the layout and structure of the GUI are analyzed to detect changes. This is a typical application of AI.

 D. Using a ML-based classifier to flag if changes in GUI are to be flagged for humans.

An ML-based classifier can intelligently determine significant changes and decide if they need human review, which is a sophisticated AI application.

\

Facial recognition applications involvecomplex computational tasks, including:

Feature Extraction- Identifying unique facial landmarks.

Model Training and Updates- Continuous learning and adaptation of user data.

Image Processing- Handling real-time image recognition under various lighting and angles.

In this scenario, themobile device is experiencing continuous restarts, which suggestsa resource overloadcaused by excessive processing demands.

Mobile devices have limited computational power.

Unlike servers, mobile devices lack powerful GPUs/TPUs required for deep learning models.

On-device learning is computationally expensive.

The model is likely performingreal-time learning, which can overwhelm the CPU and RAM.

Continuous feedback transmission may cause overheating.

If the system is running multiple processes—training, inference, and network communication—it can overload system resources and cause crashes.

(A) The feedback requires a physical connection and cannot be sent over the Internet.❌(Incorrect)

Feedback transmission over the internet is common for cloud-based AI services.This is not the cause of the issue.

(B) Mobile operating systems cannot process machine learning algorithms.❌(Incorrect)

Many mobile applications use ML models efficiently. The problem here is thehigh computational intensity, not the OS’s ability to run ML algorithms.

(C) The size of the application is consuming too much of the phone’s storage capacity.❌(Incorrect)

Storage issues typically result in installation failures or lag,not device restarts.The issue here isprocessing overload, not storage space.

AI-based applications require significant computational power."The computational intensity of AI-based applications can pose a challenge when deployed on resource-limited devices."​

Edge devices may struggle with processing complex ML workloads."Deploying AI models on mobile or edge devices requires optimization, as these devices have limited processing capabilities compared to cloud environments."​

Why is Option D Correct?Why Other Options are Incorrect?References from ISTQB Certified Tester AI Testing Study GuideThus,option D is the correct answer, as thecomputational demands of the facial recognition system are too high for the mobile hardware to handle, causing continuous restarts.

According to the ISTQB Certified Tester AI Testing (CT-AI) syllabus, data acquisition, a critical step in data preparation for machine learning (ML) workflows, consists of three key activities:

Identification:This step involves determining the types of data required for training and prediction. For example, in a self-driving car application, data types such as radar, video, laser imaging, and LiDAR (Light Detection and Ranging) data may be identified as necessary sources.

Gathering:After identifying the required data types, the sources from which the data will be collected are determined, along with the appropriate collection methods. An example could be gathering financial data from the International Monetary Fund (IMF) and integrating it into an AI-based system.

Labeling:This process involves annotating or tagging the collected data to make it meaningful for supervised learning models. Labeling is an essential activity that helps machine learning algorithms differentiate between categories and make accurate predictions.

These activities ensure that the data is suitable for training and testing machine learning models, forming the foundation of data preparation​.

Narrow AI (also known as Weak AI) is designed to perform specific tasks without possessing general intelligence or consciousness. The AI system in the question is capable of recognizing speech and responding to specific booking-related requests but fails when asked about unrelated topics (such as weather or labor markets)​.

Option A:“General AI”

Incorrect. General AI (AGI) refers to an AI system that can perform any intellectual task a human can. The described system is task-specific and does not exhibit general intelligence​.

Option B:“Narrow AI”

Correct. The AI system is limited to a predefined domain (ticket booking) and cannot process unrelated questions. This is characteristic of Narrow AI, which excels at specific tasks but lacks broader cognitive abilities​.

Option C:“Super AI”

Incorrect. Super AI surpasses human intelligence, exhibiting advanced reasoning and creativity. The AI in the scenario is far from this level​.

Option D:“Conventional AI”

Incorrect. Conventional AI is a broader term that may include rule-based systems. The described system relies on machine learning and natural language processing, making it more aligned with Narrow AI​.

Definition of Narrow AI:"Narrow AI refers to AI systems that are designed to perform a single task or a limited set of tasks, without general intelligence"​.

General vs. Narrow AI:"General AI remains an area of research, while most current AI applications fall into the category of Narrow AI"​.

Analysis of the Answer Options:ISTQB CT-AI Syllabus References:Thus,option B is the correct categorization for the AI-based ticket booking system.

 Type of Testing for Stock-Price Prediction Models:Concept drift refers to the change in the statistical properties of the target variable over time. Severe weather storms causing massive losses in coffee production and affecting stock prices would require testing for concept drift to ensure that the model adapts to new patterns in data over time.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 7.6 Testing for Concept Drift, which explains the need to test for concept drift in models that might be affected by changing external factors​​.

In AI-based systems, particularly those where theinternal structure has been generated by another software system, the complexity often makes it difficult for human testers to analyze the inner workings. As per the ISTQB Certified Tester AI Testing (CT-AI) Syllabus:

Black-box testingis particularly useful when dealing with AI systems that have been generated by another system because:

It allows testingwithout requiring knowledge of the internal logic.

The AI model may be too complex for human testers to comprehend, making white-box testing ineffective.

Black-box testing evaluates theinputs and outputs, ensuring functional correctnesswithout needing insight into how the system reaches a decision​.

Why other options are incorrect?

A (Test automation and black-box testing): While automation is possible,black-box testing is not primarily about automationbut aboutabstracting the internal complexity.

B (Understanding the logic of the software): This contradicts the premise of black-box testing, which is designed totest functionality without needing to understandthe inner workings.

C (Checking transparency of the algorithm):Black-box testing does not check algorithm transparency—that would requirewhite-box testing or explainability techniques​.

Thus, the best choice isOption D, as black-box testingremoves the need to analyze the internal structure of AI systems, making it the most appropriate testing method in this case.

Certified Tester AI Testing Study Guide References:

ISTQB CT-AI Syllabus v1.0, Section 8.5 (Challenges Testing Complex AI-Based Systems)

ISTQB CT-AI Syllabus v1.0, Section 8.6 (Testing the Transparency, Interpretability, and Explainability of AI-Based Systems)​

 AI-Based Autonomous Functions:An AI-based autonomous system is one that can respond to its environment without human intervention. The other options either involve human decisions or do not use AI at all.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Autonomy and Testing Autonomous AI-Based Systems​​​​.