Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
1.329278631s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:42:18 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.329372862s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
1.277291924s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:41:53 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.277328225s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
1.903984348s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:41:24 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.904022876s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
34.9851ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:41:23 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 35.062227ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
1.493079962s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:41:12 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.493105533s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
1.724168826s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:40:43 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.724200287s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
37.23652ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:40:40 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 37.335603ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
1.486784407s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:39:58 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.486811672s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
1.293225709s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:39:47 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.293331734s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
32.53313ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:39:39 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 32.563107ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
1.370408393s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:39:17 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.37042679s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
1.291604722s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:39:13 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.291630194s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
1.323264906s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:38:50 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.323361828s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
1.26155213s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:38:37 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.261654341s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
33.432084ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:38:37 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 33.51655ms
core
satellite
us1
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
35.609688ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:38:10 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 35.627634ms
core
satellite
us1
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
30.519032ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:37:49 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 30.547179ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
1.365511435s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:37:38 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.36553497s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
1.674912849s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:37:35 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.674939657s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
1.436486662s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:37:19 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.436525959s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
1.487062085s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:36:52 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.487197683s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
29.725591ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:36:44 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 29.808953ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
1.30123255s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:36:38 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.301295228s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
1.667690292s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:36:29 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.667720846s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
1.810540666s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:36:06 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.810576038s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
30.821079ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:35:46 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 30.852381ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
1.419625567s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:35:38 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.419749326s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
36.783111ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:35:15 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 36.812046ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
1.435241994s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:34:54 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.435335257s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
35.290392ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:34:40 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 35.32122ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.712434823s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:34:37 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 2.712551994s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
8.072827255s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:34:14 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 8.072878561s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
30.962494ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:33:48 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 30.998932ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
2.741378001s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:33:33 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.741480663s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
3.078091253s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:33:25 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 3.078124895s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
1.476326148s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:32:45 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.476355499s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
37.812707ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:32:45 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 37.84768ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
1.456117325s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:32:33 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.456153192s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
31.369538ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:32:27 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 31.395042ms
core
satellite
us1
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
35.755072ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:32:05 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 35.820522ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
1.430359401s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:31:59 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.430408987s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
1.392809327s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:31:51 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.392858218s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
1.456174634s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:31:19 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.456223024s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
36.361964ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:30:52 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 36.392837ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
1.26901611s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:30:39 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.269046892s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
1.36250636s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:30:13 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.362540898s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
32.747554ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:29:57 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 32.770767ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
1.734803702s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:29:39 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.73483389s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
1.34159195s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:29:39 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.341618455s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
48.438981ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:29:38 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 48.460774ms
core
satellite
us1
|