CTS/GTS问题分析14

由于测试手法不规范造成的cts-on-gsi无法retry的问题

问题初探

测试命令： run cts-on-gsi-retry --retry 6 -s 8981083
报错堆栈(host log中)：

02-13 11:31:51 D/BaseTestSuite: Initializing ModuleRepo
ABIs:[{arm64-v8a, bitness=64}]
Test Args:[com.android.tradefed.testtype.AndroidJUnitTest:rerun-from-file:true, com.android.tradefed.testtype.AndroidJUnitTest:fallback-to-serial-rerun:false, com.android.compatibility.testtype.LibcoreTest:rerun-from-file:true, com.android.compatibility.testtype.LibcoreTest:fallback-to-serial-rerun:false, com.android.tradefed.testtype.AndroidJUnitTest:exclude-annotation:android.platform.test.annotations.RestrictedBuildTest, com.android.tradefed.testtype.HostTest:exclude-annotation:android.platform.test.annotations.RestrictedBuildTest, com.android.compatibility.common.tradefed.testtype.JarHostTest:exclude-annotation:android.platform.test.annotations.RestrictedBuildTest, com.android.tradefed.testtype.AndroidJUnitTest:exclude-annotation:android.platform.test.annotations.AppModeInstant, com.android.compatibility.common.tradefed.testtype.JarHostTest:exclude-annotation:android.platform.test.annotations.AppModeInstant, com.android.tradefed.testtype.HostTest:exclude-annotation:android.platform.test.annotations.AppModeInstant]
Module Args:[]
Includes:{armeabi-v7a CtsWindowManagerDeviceTestCases=[armeabi-v7a CtsWindowManagerDeviceTestCases android.server.wm.LocationOnScreenTests#testLocationOnDisplay_appWindow, armeabi-v7a CtsWindowManagerDeviceTestCases android.server.wm.LocationOnScreenTests#testLocationOnDisplay_appWindow_fullscreen, armeabi-v7a CtsWindowManagerDeviceTestCases android.server.wm.LocationOnScreenTests#testLocationOnDisplay_appWindow_displayCutoutNever, armeabi-v7a CtsWindowManagerDeviceTestCases android.server.wm.LocationOnScreenTests#testLocationOnDisplay_floatingWindow, armeabi-v7a CtsWindowManagerDeviceTestCases android.server.wm.LocationOnScreenTests#testLocationOnDisplay_appWindow_displayCutoutShortEdges], armeabi-v7a CtsAppSecurityHostTestCases=[armeabi-v7a CtsAppSecurityHostTestCases android.appsecurity.cts.AdoptableHostTest#testApps], armeabi-v7a CtsDevicePolicyManagerTestCases=[armeabi-v7a CtsDevicePolicyManagerTestCases com.android.cts.devicepolicy.DeviceOwnerTest#testLockTaskAfterReboot_deviceOwnerUser], armeabi-v7a CtsAccessibilityServiceTestCases=[armeabi-v7a CtsAccessibilityServiceTestCases], armeabi-v7a CtsInputMethodServiceHostTestCases=[armeabi-v7a CtsInputMethodServiceHostTestCases android.inputmethodservice.cts.hostside.InputMethodServiceLifecycleTest#testInputUnbindsOnImeStoppedFull], armeabi-v7a CtsJobSchedulerTestCases=[armeabi-v7a CtsJobSchedulerTestCases android.jobscheduler.cts.TriggerContentTest#testPhotoAdded_Reschedule], armeabi-v7a CtsNetTestCases=[armeabi-v7a CtsNetTestCases android.net.cts.DnsTest#testDnsWorks], armeabi-v7a CtsCarrierApiTestCases=[armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.ApnDatabaseTest#testUpdateConflictCase, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testCarrierConfigIsAccessible, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.ApnDatabaseTest#testValidCase, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testVoicemailTableIsAccessible, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.ApnDatabaseTest#testQueryConflictCase, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.ApnDatabaseTest#testDeleteConflictCase, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testTelephonyApisAreAccessible, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.NetworkScanApiTest#testRequestNetworkScan, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testSubscriptionInfoListing, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testVoicemailStatusTableIsAccessible, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testPhoneStateListener], armeabi-v7a CtsLocationTestCases=[armeabi-v7a CtsLocationTestCases android.location.cts.GnssPseudorangeVerificationTest#testPseudorangeValue, armeabi-v7a CtsLocationTestCases android.location.cts.GnssPseudorangeVerificationTest#testPseudoPosition, armeabi-v7a CtsLocationTestCases android.location.cts.GnssStatusTest#testGnssStatusChanges, armeabi-v7a CtsLocationTestCases android.location.cts.GnssLocationValuesTest#testAccuracyFields, armeabi-v7a CtsLocationTestCases android.location.cts.GnssLocationRateChangeTest#testVariedRatesRepetitive, armeabi-v7a CtsLocationTestCases android.location.cts.GnssMeasurementsConstellationTest#testGnssMultiConstellationSupported, armeabi-v7a CtsLocationTestCases android.location.cts.GnssLocationRateChangeTest#testVariedRatesOnOff, armeabi-v7a CtsLocationTestCases android.location.cts.GnssStatusTest#testGnssStatusValues, armeabi-v7a CtsLocationTestCases android.location.cts.GnssMeasurementValuesTest#testListenForGnssMeasurements, armeabi-v7a CtsLocationTestCases android.location.cts.GnssLocationRateChangeTest#testVariedRates, armeabi-v7a CtsLocationTestCases android.location.cts.GnssMeasurementWhenNoLocationTest#testGnssMeasurementWhenNoLocation, armeabi-v7a CtsLocationTestCases android.location.cts.GnssLocationValuesTest#testLocationRegularFields], armeabi-v7a CtsSyncContentHostTestCases=[armeabi-v7a CtsSyncContentHostTestCases], armeabi-v7a CtsStatsdHostTestCases=[armeabi-v7a CtsStatsdHostTestCases android.cts.statsd.atom.UidAtomTests#testCpuTimePerUid], armeabi-v7a CtsSecurityHostTestCases=[armeabi-v7a CtsSecurityHostTestCases android.security.cts.KernelConfigTest#testConfigROData], armeabi-v7a CtsIncidentHostTestCases=[armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.IncidentdTest#testIncidentReportDumpExplicit, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.BatteryStatsValidationTest#testCachedDuration, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.IncidentdTest#testIncidentReportDumpLocal, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.BatteryStatsValidationTest#testGpsUpdates, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.BatteryStatsValidationTest#testRealtime, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.BatteryStatsValidationTest#testJobBgVsFg, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.BatteryStatsValidationTest#testSyncBgVsFg], armeabi-v7a CtsSliceTestCases=[armeabi-v7a CtsSliceTestCases android.slice.cts.SliceMetricsTest#testLogVisible], armeabi-v7a CtsSecurityTestCases=[armeabi-v7a CtsSecurityTestCases android.security.cts.MotionEventTest#testActionOutsideDoesNotContainedObscuredInformation], armeabi-v7a CtsLibcoreTestCases=[armeabi-v7a CtsLibcoreTestCases libcore.java.nio.channels.DatagramChannelMulticastTest#test_multicastLoopOption_IPv6, armeabi-v7a CtsLibcoreTestCases libcore.java.net.SocketTest#testSocketTestAllAddresses, armeabi-v7a CtsLibcoreTestCases libcore.java.net.InetAddressTest#test_getByName_invalid, armeabi-v7a CtsLibcoreTestCases libcore.libcore.icu.DateIntervalFormatTest#testEndAtMidnight], armeabi-v7a CtsCameraTestCases=[armeabi-v7a CtsCameraTestCases android.hardware.cts.CameraGLTest#testCameraToSurfaceTextureMetadata, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testTakePictureZsl, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testJpegExif, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testAllocateBitmap, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testTakePicture, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testAfRegions, armeabi-v7a CtsCameraTestCases android.hardware.cts.CameraTest#testPreviewFpsRange, armeabi-v7a CtsCameraTestCases android.hardware.cts.CameraTest#testPreviewPictureSizesCombination, armeabi-v7a CtsCameraTestCases android.hardware.cts.CameraTest#testImmediateZoom, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.ImageReaderTest#testAllOutputYUVResolutions, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testAeRegions], armeabi-v7a CtsOsTestCases=[armeabi-v7a CtsOsTestCases android.os.storage.cts.StorageManagerTest#testMountAndUnmountTwoObbs, armeabi-v7a CtsOsTestCases android.os.cts.ParcelTest#testMaliciousMapWrite, armeabi-v7a CtsOsTestCases android.os.storage.cts.StorageManagerTest#testMountAndUnmountObbNormal], armeabi-v7a CtsSkQPTestCases=[armeabi-v7a CtsSkQPTestCases]}
Excludes:{armeabi-v7a CtsJvmtiRunTest1906HostTestCases=[armeabi-v7a CtsJvmtiRunTest1906HostTestCases], ... }
02-13 11:31:55 D/CompatibilityTestSuite: No module that needed to run in retry were found. nothing to do.
02-13 11:31:55 I/ITestSuite: No config were loaded. Nothing to run.
02-13 11:31:55 I/ITestSuite: No config were loaded. Nothing to run.
02-13 11:31:55 I/ITestSuite: No tests to be run.

从堆栈上看是正常的，需要再深入看一下

问题分析

首先从关键的一句log入手

No module that needed to run in retry were found. nothing to do.

public LinkedHashMap<String, IConfiguration> loadingStrategy(Set<IAbi> abis, File testsDir, String suitePrefix, String suiteTag)
 {
   LinkedHashMap<String, IConfiguration> loadedConfigs = super.loadingStrategy(abis, testsDir, suitePrefix, suiteTag);
   if (loadedConfigs.size() == 0) {
     if (this.mIsRetry) {
       LogUtil.CLog.logAndDisplay(Log.LogLevel.DEBUG, "No module that needed to run in retry were found. nothing to do.");
     } else {
       throw new IllegalArgumentException(String.format("No config files found in %s or in resources.", new Object[] {testsDir
         .getAbsolutePath() }));
     }
   }
   return loadedConfigs;
 }

很明显是loadingStrategy返回的LinkedHashMap为空导致的问题，那么需要看一下为什么返回值是空

看BaseTestSuite中的loadingStrategy，这里返回null

public LinkedHashMap<String, IConfiguration> loadingStrategy(Set<IAbi> abis, File testsDir, String suitePrefix, String suiteTag)
{
  LinkedHashMap<String, IConfiguration> loadedConfigs = new LinkedHashMap();
  if (!this.mSkipJarLoading) {
    loadedConfigs.putAll(
      getModuleLoader().loadConfigsFromJars(abis, suitePrefix, suiteTag));
  }
  if (this.mConfigPatterns.isEmpty())
  {
    this.mConfigPatterns.add(".*\\.config");
    this.mConfigPatterns.add(".*\\.xml");
  }
  loadedConfigs.putAll(
    getModuleLoader()
    .loadConfigsFromDirectory(testsDir, abis, suitePrefix, suiteTag, this.mConfigPatterns));
  return loadedConfigs;
}

这个函数是为了加载我们给测试项提供的额外的配置文件，一般在google/vts/vts9.0_r5/android-vts/testcases的文件下，如BionicUnitTestsStatic.config等等，从中读取配置文件；retry时其调用栈为：

vts-tf > java.lang.RuntimeException
    at com.android.tradefed.testtype.suite.BaseTestSuite.loadingStrategy(BaseTestSuite.java:91)
    at com.android.compatibility.common.tradefed.testtype.suite.CompatibilityTestSuite.loadingStrategy(CompatibilityTestSuite.java:164)
    at com.android.tradefed.testtype.suite.BaseTestSuite.loadTests(BaseTestSuite.java:76)
    at com.android.compatibility.common.tradefed.testtype.suite.CompatibilityTestSuite.loadTests(CompatibilityTestSuite.java:113)
    at com.android.tradefed.testtype.suite.ITestSuite.loadAndFilter(ITestSuite.java:262)
    at com.android.tradefed.testtype.suite.ITestSuite.createExecutionList(ITestSuite.java:304)
    at com.android.tradefed.testtype.suite.ITestSuite.run(ITestSuite.java:367)
    at com.android.compatibility.common.tradefed.testtype.retry.RetryFactoryTest.run(RetryFactoryTest.java:177)
    at com.android.tradefed.invoker.InvocationExecution.runTests(InvocationExecution.java:384)
    at com.android.tradefed.invoker.TestInvocation.prepareAndRun(TestInvocation.java:358)
    at com.android.tradefed.invoker.TestInvocation.performInvocation(TestInvocation.java:205)
    at com.android.tradefed.invoker.TestInvocation.invoke(TestInvocation.java:667)
    at com.android.tradefed.command.CommandScheduler$InvocationThread.run(CommandScheduler.java:566)

继续沿着栈查看

public LinkedHashMap<String, IConfiguration> loadConfigsFromDirectory(File testsDir, Set<IAbi> abis, String suitePrefix, String suiteTag, List<String> patterns)
{
  LinkedHashMap<String, IConfiguration> toRun = new LinkedHashMap();
   
  List<File> listConfigFiles = new ArrayList();
  List<File> extraTestCasesDirs = Arrays.asList(new File[] { testsDir });
  listConfigFiles.addAll(
    ConfigurationUtil.getConfigNamesFileFromDirs(suitePrefix, extraTestCasesDirs, patterns));
   
  Collections.sort(listConfigFiles);
  for (File configFile : listConfigFiles) {
    toRun.putAll(
      loadOneConfig(configFile
      .getName(), configFile.getAbsolutePath(), abis, suiteTag));
  }
  return toRun;
}

可见有两种可能 1. listConfigFiles为空 2. 循环loadOneConfig一直没有加载config

若第一种情况，那么只能是测试环境有问题，配置文件完全被删了，看了下排除该情况；那么只能是第二种情况

查看SuiteModuleLoader中的相应逻辑

private LinkedHashMap<String, IConfiguration> loadOneConfig(String configName, String configFullName, Set<IAbi> abis, String suiteTag)
{
  LinkedHashMap<String, IConfiguration> toRun = new LinkedHashMap();
  String name = configName.replace(".config", "");
  String[] pathArg = { configFullName };
  try
  {
    for (IAbi abi : abis)
    {
      String id = AbiUtils.createId(abi.getName(), name);
      if (shouldRunModule(id))
      {
        IConfiguration config = this.mConfigFactory.createConfigurationFromArgs(pathArg);
        if (!Strings.isNullOrEmpty(suiteTag)) {
          if (!config.getConfigurationDescription().getSuiteTags().contains(suiteTag))
          {
            LogUtil.CLog.d("Configuration %s does not include the suite-tag '%s'. Ignoring it.", new Object[] { configFullName, suiteTag });
            continue;
          }
        }
        List<ConfigurationDef.OptionDef> optionsToInject = new ArrayList();
        if (this.mModuleOptions.containsKey(name)) {
          optionsToInject.addAll((Collection)this.mModuleOptions.get(name));
        }
        if (this.mModuleOptions.containsKey(id)) {
          optionsToInject.addAll((Collection)this.mModuleOptions.get(id));
        }
        config.injectOptionValues(optionsToInject);
         
        ITargetPreparer preparer;
        List<ITargetPreparer> preparers = config.getTargetPreparers();
        for (Iterator localIterator2 = preparers.iterator(); localIterator2.hasNext();)
        {
          preparer = (ITargetPreparer)localIterator2.next();
          if ((preparer instanceof IAbiReceiver)) {
            ((IAbiReceiver)preparer).setAbi(abi);
          }
        }
         
        Object tests = config.getTests();
        for (IRemoteTest test : (List<IRemoteTest>)tests)
        {
          String className = test.getClass().getName();
          if (this.mTestOptions.containsKey(className)) {
            config.injectOptionValues((List)this.mTestOptions.get(className));
          }
          addFiltersToTest(test, abi, name, this.mIncludeFilters, this.mExcludeFilters);
          if ((test instanceof IAbiReceiver)) {
            ((IAbiReceiver)test).setAbi(abi);
          }
        }
        config.getConfigurationDescription().setAbi(abi);
        config.getConfigurationDescription().setModuleName(name);
        toRun.put(id, config);
      }
    }
  }
  catch (ConfigurationException e)
  {
    throw new RuntimeException(String.format("Error parsing configuration: %s: '%s'", new Object[] { configFullName, e
     
      .getMessage() }), e);
  }
  return toRun;
}

private boolean shouldRunModule(String moduleId)
{
  List<SuiteTestFilter> mdIncludes = getFilterList(this.mIncludeFilters, moduleId);
  List<SuiteTestFilter> mdExcludes = getFilterList(this.mExcludeFilters, moduleId);
  return ((this.mIncludeAll) || (!mdIncludes.isEmpty())) && (!containsModuleExclude(mdExcludes));
}

我们注意到在loadOneConfig中有个过滤条件，shouldRunModule；那么在retry时这里的逻辑是过滤出要被执行的module的相应的配置属性

那么为什么这个值一直是false呢，mIncludeFilters本身不是null（host log中打印出来了），那么只能getFilterList(this.mIncludeFilters, moduleId)得到null；

moduleId为abi和module name的组合，到这里，我们就怀疑是abi造成的问题了；再回头看host log；发现果然：

ABIs:[{arm64-v8a, bitness=64}]
Includes:{armeabi-v7a CtsWindowManagerDeviceTestCases=[armeabi-v7a CtsWindowManagerDeviceTestCases ...

由于abi的不匹配造成的问题，retry时查询要执行的module的config时，由于abi不匹配造成测试框架误判断shouldRunModule = false；

问题总结

1.测试手法的问题，cts-og-gsi默认只测64位，但是如果后面加上-a armeabi-v7a的话，也能进行32位的测试；测试第一次测试时加了-a armeabi-v7a；但是后面进行retry时没有加，导致框架默认abi为arm64-v8a；导致后续retry流程module因为abi的关系无法匹配；读取要retry的报告为v7a，加载config时默认v8a导致的问题， 请测试同学注意下；测cts-on-gsi时后面就不用加-a了，避免该问题重复发生

2.框架有轻微的问题，但本质来说是测试手法问题，google不会认的