Back

Establish, implement, and maintain system testing procedures.


CONTROL ID
11744
CONTROL TYPE
Establish/Maintain Documentation
CLASSIFICATION
Preventive

SUPPORTING AND SUPPORTED CONTROLS




This Control directly supports the implied Control(s):
  • Perform Quality Management on all newly developed or modified systems., CC ID: 01100

This Control has the following implementation support Control(s):
  • Restrict production data from being used in the test environment., CC ID: 01103
  • Protect test data in the development environment., CC ID: 12014
  • Control the test data used in the development environment., CC ID: 12013
  • Select the test data carefully., CC ID: 12011
  • Test all software changes before promoting the system to a production environment., CC ID: 01106
  • Test security functionality during the development process., CC ID: 12015
  • Include system performance in the scope of system testing., CC ID: 12624
  • Include security controls in the scope of system testing., CC ID: 12623
  • Include business logic in the scope of system testing., CC ID: 12622
  • Review and test custom code to identify potential coding vulnerabilities., CC ID: 01316
  • Disseminate and communicate the system testing procedures to interested personnel and affected parties., CC ID: 15471
  • Establish, implement, and maintain poor quality material removal procedures., CC ID: 06214


SELECTED AUTHORITY DOCUMENTS COMPLIED WITH




  • App 2-1 Item Number III.4(3): Program source code must be assessed and evaluated and the results recorded and stored. This is a control item that constitutes a relatively small risk to financial information. This is an IT general control. App 2-1 Item Number III.4(4): Important programs must be test… (App 2-1 Item Number III.4(3), App 2-1 Item Number III.4(4), App 2-1 Item Number III.5(4), App 2-1 Item Number III.5(6), App 2-1 Item Number V.4(1) thru App 2-1 Item Number V.4(4), Appendix 1 Correspondence of the System Management Standards - Supplementary Edition to other standards)
  • O68: The organization shall establish test environments that do not affect production environments. O68.2(1).3: The organization should avoid testing when the production systems are operating, when possible. If it is unavoidable, potential testing effects should be made known and test limitations sh… (O68, O68.2(1).3, T11.2(2), FISC Security Guidelines on Computer Systems for Banking and Related Financial Institutions, 7th Edition)
  • A methodology for system testing should be established. The scope of testing should cover business logic, system function, security controls and system performance under various load and stress conditions. A test plan should be established and approved before testing. (§ 5.7.1, Technology Risk Management Guidelines, January 2021)
  • An independent party, as well as the developer, should test or review the software. (Control: 0403, Australian Government Information Security Manual: Controls)
  • The testing of the high-risk AI systems shall be performed, as appropriate, at any point in time throughout the development process, and, in any event, prior to the placing on the market or the putting into service. Testing shall be made against preliminarily defined metrics and probabilistic thresh… (Article 9 7., Proposal for a Regulation of The European Parliament and of The Council Laying Down Harmonized Rules On Artificial Intelligence (Artificial Intelligence Act) and Ameding Certain Union Legislative Acts)
  • The continued confidentiality, completeness, integrity and availability of the entity's systems and back-up information is evaluated and confirmed on a periodic basis. (S7.5 Testing confidentiality, completeness, integrity and availability of systems and back-up data, Privacy Management Framework, Updated March 1, 2020)
  • Documented. (11.1.1 Bullet 1, Payment Card Industry Data Security Standard Requirements and Testing Procedures, Defined Approach Requirements, Version 4.0)
  • Kept up to date. (11.1.1 Bullet 2, Payment Card Industry Data Security Standard Requirements and Testing Procedures, Defined Approach Requirements, Version 4.0)
  • In use. (11.1.1 Bullet 3, Payment Card Industry Data Security Standard Requirements and Testing Procedures, Defined Approach Requirements, Version 4.0)
  • Kept up to date. (11.1.1 Bullet 2, Self-Assessment Questionnaire D for Merchants and Attestation of Compliance for use with PCI DSS Version 4.0)
  • In use. (11.1.1 Bullet 3, Self-Assessment Questionnaire D for Merchants and Attestation of Compliance for use with PCI DSS Version 4.0)
  • Documented. (11.1.1 Bullet 1, Self-Assessment Questionnaire D for Merchants and Attestation of Compliance for use with PCI DSS Version 4.0)
  • Documented. (11.1.1 Bullet 1, Self-Assessment Questionnaire D for Service Providers and Attestation of Compliance for use with PCI DSS Version 4.0)
  • Kept up to date. (11.1.1 Bullet 2, Self-Assessment Questionnaire D for Service Providers and Attestation of Compliance for use with PCI DSS Version 4.0)
  • In use. (11.1.1 Bullet 3, Self-Assessment Questionnaire D for Service Providers and Attestation of Compliance for use with PCI DSS Version 4.0)
  • Quality Assurance of the system development methodology should include confirming that individuals responsible for testing systems under development are following the methodology. (CF.17.03.02d-2, The Standard of Good Practice for Information Security)
  • Before critical desktop applications are made available to users, checks should be performed to ensure that they can be supported on a continuing basis (e.g., by an individual or group of individuals skilled in developing desktop applications). (CF.13.04.10, The Standard of Good Practice for Information Security)
  • Standards / procedures for testing systems under development should cover the types of hardware, software, and services to be tested. (CF.18.04.02, The Standard of Good Practice for Information Security)
  • Tests should involve the system running in expected conditions. (CF.18.04.07, The Standard of Good Practice for Information Security)
  • Quality Assurance of the system development methodology should include confirming that individuals responsible for testing systems under development are following the methodology. (CF.17.03.02d-2, The Standard of Good Practice for Information Security, 2013)
  • Before critical desktop applications are made available to users, checks should be performed to ensure that they can be supported on a continuing basis (e.g., by an individual or group of individuals skilled in developing desktop applications). (CF.13.04.10, The Standard of Good Practice for Information Security, 2013)
  • Standards / procedures for testing systems under development should cover the types of hardware, software, and services to be tested. (CF.18.04.02, The Standard of Good Practice for Information Security, 2013)
  • System testing should be performed independently of system development staff, involve business users, and simulate the live environment. (CF.18.04.05, The Standard of Good Practice for Information Security, 2013)
  • Tests should involve the system running in expected conditions. (CF.18.04.07, The Standard of Good Practice for Information Security, 2013)
  • The organization shall ensure operators, equipment, and facilities are ready to conduct verifications. (§ 6.4.6.3(b)(1), ISO 15288-2008 Systems and software engineering - System life cycle processes, R 2008)
  • Test documentation should be developed and should include the following: a list of the security functions to be tested; test plans; descriptions of the test procedures; the scenarios to be used for testing each security function; what the expected test results should be; and what the actual results … (§ 18.3, ISO 15408-3 Common Criteria for Information Technology Security Evaluation Part 3, 2008)
  • If the organization decides to purchase products instead of developing them, a formal testing and acquisition process should be implemented. If the security functionality in the product does not meet the organization's requirements, the purchase should not be made or the risk should be assessed to d… (§ 12.1.1, ISO 27002 Code of practice for information security management, 2005)
  • For software systems assigned to Class B and Class C software safety classes, the medical device manufacturer shall develop methods, strategies, and procedures to verify each software unit. If the verification is accomplished via testing, the manufacturer shall evaluate the test procedures to ensure… (§ 5.5.2, ISO 62304 - 2006 Medical device software - Software life cycle processes, 2006)
  • Security testing processes should be defined and implemented in the development life cycle. (§ 8.29 Control, ISO/IEC 27002:2022, Information security, cybersecurity and privacy protection — Information security controls, Third Edition)
  • Embedded devices shall provide active monitoring of the device's diagnostic and test interface(s) and generate an audit log entry when attempts to access these interface(s) are detected. (13.3.3 (1) ¶ 1, Security for Industrial Automation and Control Systems, Part 4-2: Technical Security Requirements for IACS components)
  • Does the testing of Internet facing software and infrastructure prior to implementation include issue tracking and resolution? (§ I.2.21.1, Shared Assessments Standardized Information Gathering Questionnaire - I. Information Systems Acquisition Development & Maintenance, 7.0)
  • The organization must develop a comprehensive set of test transactions and data that represents the various conditions and activities that will likely be encountered during processing. (CSR 6.3.10, Pub 100-17 Medicare Business Partners Systems Security, Transmittal 7, Appendix A: CMS Core Security Requirements CSR, March 17, 2006)
  • shall include testing of management, operational, and technical controls of every information system identified in the inventory required under section 3505(c); (§ 3554(b)(5)(A), Federal Information Security Modernization Act of 2014)
  • shall include using automated tools, consistent with standards and guidelines promulgated under section 11331 of title 40; (§ 3554(b)(5)(C), Federal Information Security Modernization Act of 2014)
  • testing of the effectiveness of information security policies, procedures, and practices of a representative subset of the agency's information systems; (§ 3555(a)(2)(A), Federal Information Security Modernization Act of 2014)
  • An independent software evaluation should be used, especially for higher risk applications. (§ 4.9, General Principles of Software Validation; Final Guidance for Industry and FDA Staff, Version 2.0)
  • Software testing on a medical device or a component of a medical device that is being tested on humans before federal drug administration clearance may require an investigational device exemption or institutional review board approval. (§ 5.2.5 ¶ 16, General Principles of Software Validation; Final Guidance for Industry and FDA Staff, Version 2.0)
  • Test procedures, test data, and test results should allow for objective pass or fail decisions to be reached. (§ 5.2.5 ¶ 17, General Principles of Software Validation; Final Guidance for Industry and FDA Staff, Version 2.0)
  • The level of the validation effort should be commensurate with the risk of the automated operations. (§ 6.1 ¶ 1, General Principles of Software Validation; Final Guidance for Industry and FDA Staff, Version 2.0)
  • The test results should be documented and analyzed to determine if the software is validated for its intended use. (§ 6.2 ¶ 3, General Principles of Software Validation; Final Guidance for Industry and FDA Staff, Version 2.0)
  • Systems, applications, and data recovery is tested at least annually. (Domain 5: Assessment Factor: Resillience Planning and Strategy, TESTING Baseline 2 ¶ 3, FFIEC Cybersecurity Assessment Tool, Baseline, May 2017)
  • Operational testing. (App A Objective 2:4a Bullet 3, FFIEC Information Technology Examination Handbook - Architecture, Infrastructure, and Operations, June 2021)
  • Database administration, systems analysis, client support, systems administration, and network administration. (App A Objective 2:9c Bullet 8, FFIEC Information Technology Examination Handbook - Architecture, Infrastructure, and Operations, June 2021)
  • Develops test scripts and implementation plans. (App A Objective 6.11.f, FFIEC Information Technology Examination Handbook - Information Security, September 2016)
  • The organization should have a testing approach. The testing team should test the functions, security, and internal controls features to ensure they are operating properly; test the components to ensure they interact correctly; and test the application to ensure it meets the defined acceptance crite… (Pg 29, Pg 30, FFIEC IT Examination Handbook - Development and Acquisition)
  • The organization should test new systems and products before deployment. The testing process should verify that the new systems or software operate with the current systems or software installed on the organization's system. (Pg 31, FFIEC IT Examination Handbook - Management)
  • Calls for testing of new systems and applications, including acceptance testing, and requires testing not use production data to preserve confidentiality. (CC-2, Federal Information System Controls Audit Manual (FISCAM), February 2009)
  • Are there implemented policies, procedures, and practices that include unit testing, system testing, integration testing, and acceptance testing for all new systems or modified systems? (IT - Networks Q 39, Automated Integrated Regulatory Examination System (AIRES) IT Exam Questionnaires, version 073106A)
  • At the following level of rigor: [Assignment: organization-defined breadth and depth of criticality analysis]. (SA-15(3) ¶ 1(b), Control Baselines for Information Systems and Organizations, NIST SP 800-53B, High Impact Baseline, October 2020)
  • At the following level of rigor: [Assignment: organization-defined breadth and depth of criticality analysis]. (SA-15(3) ¶ 1(b), Control Baselines for Information Systems and Organizations, NIST SP 800-53B, Moderate Impact Baseline, October 2020)
  • At the following level of rigor: [Assignment: organization-defined breadth and depth of criticality analysis]. (SA-15(3) ¶ 1(b), Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations, NIST Special Publication 800-161, Revision 1, Appendix A, C-SCRM Level 2 Controls)
  • Analyze [Assignment: organization-defined systems or systems components] supporting mission essential services or functions to ensure that the information resources are being used consistent with their intended purpose. (PM-32 Control, Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations, NIST Special Publication 800-161, Revision 1, Appendix A, C-SCRM Level 2 Controls)
  • At the following level of rigor: [Assignment: organization-defined breadth and depth of criticality analysis]. (SA-15(3) ¶ 1(b), Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations, NIST Special Publication 800-161, Revision 1, Appendix A, C-SCRM Level 3 Controls)
  • Analyze [Assignment: organization-defined systems or systems components] supporting mission essential services or functions to ensure that the information resources are being used consistent with their intended purpose. (PM-32 Control, Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations, NIST Special Publication 800-161, Revision 1, Appendix A, C-SCRM Level 3 Controls)
  • Organizational records and documents should be examined to ensure security tests and evaluations are performed on any system under development; the test results are documented and included in the Plan of Action and Milestones; the developer tests the system on a regular basis; and specific responsib… (SA-11, Guide for Assessing the Security Controls in Federal Information Systems, NIST SP 800-53A)
  • Develop and direct system testing and validation procedures and documentation. (T0061, National Initiative for Cybersecurity Education (NICE) Cybersecurity Workforce Framework, NIST Special Publication 800-181)
  • Implement new system design procedures, test procedures, and quality standards. (T0121, National Initiative for Cybersecurity Education (NICE) Cybersecurity Workforce Framework, NIST Special Publication 800-181)
  • Provide an accurate technical evaluation of the software application, system, or network, documenting the security posture, capabilities, and vulnerabilities against relevant cybersecurity compliances. (T0197, National Initiative for Cybersecurity Education (NICE) Cybersecurity Workforce Framework, NIST Special Publication 800-181)
  • Develop software system testing and validation procedures, programming, and documentation. (T0455, National Initiative for Cybersecurity Education (NICE) Cybersecurity Workforce Framework, NIST Special Publication 800-181)
  • Develop system testing and validation procedures, programming, and documentation. (T0457, National Initiative for Cybersecurity Education (NICE) Cybersecurity Workforce Framework, NIST Special Publication 800-181)
  • Perform operational testing. (T0513, National Initiative for Cybersecurity Education (NICE) Cybersecurity Workforce Framework, NIST Special Publication 800-181)
  • The organization should require system developers and integrators to perform vulnerability analysis to document vulnerabilities, exploitation potential, and risk mitigations. (App F § SA-11(2), Recommended Security Controls for Federal Information Systems, NIST SP 800-53)
  • Develop and direct system testing and validation procedures and documentation. (T0061, Reference Spreadsheet for the Workforce Framework for Cybersecurity (NICE Framework)”, July 7, 2020)
  • Implement new system design procedures, test procedures, and quality standards. (T0121, Reference Spreadsheet for the Workforce Framework for Cybersecurity (NICE Framework)”, July 7, 2020)
  • Perform operational testing. (T0513, Reference Spreadsheet for the Workforce Framework for Cybersecurity (NICE Framework)”, July 7, 2020)
  • Develop software system testing and validation procedures, programming, and documentation. (T0455, Reference Spreadsheet for the Workforce Framework for Cybersecurity (NICE Framework)”, July 7, 2020)
  • Develop system testing and validation procedures, programming, and documentation. (T0457, Reference Spreadsheet for the Workforce Framework for Cybersecurity (NICE Framework)”, July 7, 2020)
  • Provide an accurate technical evaluation of the software application, system, or network, documenting the security posture, capabilities, and vulnerabilities against relevant cybersecurity compliances. (T0197, Reference Spreadsheet for the Workforce Framework for Cybersecurity (NICE Framework)”, July 7, 2020)
  • The organization requires the developer of the information system, system component, or information system service to employ tools for comparing newly generated versions of security-relevant hardware descriptions and software/firmware source and object code with previous versions. (SA-10(4), Security and Privacy Controls for Federal Information Systems and Organizations, NIST SP 800-53, Deprecated, Revision 4, Deprecated)
  • The organization requires the developer of the information system, system component, or information system service to determine the exploitation potential for discovered vulnerabilities. (SA-15(7)(b), Security and Privacy Controls for Federal Information Systems and Organizations, NIST SP 800-53, Deprecated, Revision 4, Deprecated)
  • The organization requires the developer of the information system, system component, or information system service to structure security-relevant hardware, software, and firmware to facilitate testing. (SA-17(6), Security and Privacy Controls for Federal Information Systems and Organizations, NIST SP 800-53, Deprecated, Revision 4, Deprecated)
  • The organization approves, documents, and controls the use of live data in development and test environments for the information system, system component, or information system service. (SA-15(9) ¶ 1, Security and Privacy Controls for Federal Information Systems and Organizations, NIST SP 800-53, Revision 4)
  • At the following level of rigor: [Assignment: organization-defined breadth and depth of criticality analysis]. (SA-15(3) ¶ 1(b), Security and Privacy Controls for Information Systems and Organizations, NIST SP 800-53, Revision 5)
  • Approve, document, and control the use of live data in preproduction environments for the system, system component, or system service; and (SA-3(2)(a), Security and Privacy Controls for Information Systems and Organizations, NIST SP 800-53, Revision 5)
  • Analyze [Assignment: organization-defined systems or systems components] supporting mission essential services or functions to ensure that the information resources are being used consistent with their intended purpose. (PM-32 Control, Security and Privacy Controls for Information Systems and Organizations, NIST SP 800-53, Revision 5)
  • Bank management should thoroughly test new technology systems and products. Testing validates that equipment and systems function properly and produce the desired results. As part of the testing process, management should verify whether new technology systems operate effectively with the bank's olde… (¶ 37, Technology Risk Management Guide for Bank Examiners - OCC Bulletin 98-3)