Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 215.329485ms FAIL (2025-10-27 13:15:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 215.357467ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XX...X..........................................] duration: 408.984746ms FAIL (2025-10-27 13:14:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 409.032052ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 221.96052ms FAIL (2025-10-27 13:14:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 222.080041ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............XX...X.........................................] duration: 361.092856ms FAIL (2025-10-27 13:14:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 361.125102ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 198.643081ms FAIL (2025-10-27 13:13:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 198.676708ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............XX...X........................................] duration: 367.067309ms FAIL (2025-10-27 13:13:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 367.095712ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............XX...X.......................................] duration: 349.869473ms FAIL (2025-10-27 13:13:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 349.926059ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 209.401279ms FAIL (2025-10-27 13:12:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 209.457553ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 206.410592ms FAIL (2025-10-27 13:12:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 206.441133ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XX...X......................................] duration: 426.551407ms FAIL (2025-10-27 13:12:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 426.588848ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XX...X.....................................] duration: 370.622026ms FAIL (2025-10-27 13:11:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 370.653234ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 279.053365ms FAIL (2025-10-27 13:11:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 279.112426ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................XX...X....................................] duration: 432.117992ms FAIL (2025-10-27 13:11:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 432.143455ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 202.918103ms FAIL (2025-10-27 13:11:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 203.074335ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 177.914727ms FAIL (2025-10-27 13:10:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 177.951109ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................XX...X...................................] duration: 382.118176ms FAIL (2025-10-27 13:10:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 382.153297ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 196.175244ms FAIL (2025-10-27 13:10:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 196.204153ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................XX...X..................................] duration: 371.030282ms FAIL (2025-10-27 13:10:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 371.062449ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XX...X.................................] duration: 382.636975ms FAIL (2025-10-27 13:09:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 382.670355ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 219.825469ms FAIL (2025-10-27 13:09:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 219.856292ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XX...X................................] duration: 425.722944ms FAIL (2025-10-27 13:08:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 425.757426ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XX...X...............................] duration: 398.101695ms FAIL (2025-10-27 13:08:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 398.136529ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 220.612304ms FAIL (2025-10-27 13:08:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 220.684697ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 192.936221ms FAIL (2025-10-27 13:07:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 192.967317ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 206.06066ms FAIL (2025-10-27 13:07:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 206.09288ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................XX...X..............................] duration: 397.625592ms FAIL (2025-10-27 13:07:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 397.755648ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 212.613881ms FAIL (2025-10-27 13:06:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 212.645958ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XX...X.............................] duration: 389.049751ms FAIL (2025-10-27 13:05:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 389.085552ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 223.527885ms FAIL (2025-10-27 13:05:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 223.578067ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 197.151983ms FAIL (2025-10-27 13:05:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 197.25123ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XX...X............................] duration: 388.225106ms FAIL (2025-10-27 13:05:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 388.312099ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 190.064089ms FAIL (2025-10-27 13:04:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 190.153924ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XX...X...........................] duration: 370.250743ms FAIL (2025-10-27 13:04:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 370.361436ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 187.235172ms FAIL (2025-10-27 13:03:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 187.336669ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XX...X..........................] duration: 401.638076ms FAIL (2025-10-27 13:03:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 401.78818ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 208.007182ms FAIL (2025-10-27 13:03:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 208.039591ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XX...X.........................] duration: 401.12304ms FAIL (2025-10-27 13:02:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 401.160571ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 190.240205ms FAIL (2025-10-27 13:02:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 190.269772ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XX...X........................] duration: 422.040696ms FAIL (2025-10-27 13:02:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 422.076122ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................XX...X.......................] duration: 418.245042ms FAIL (2025-10-27 13:01:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 418.331546ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 198.280644ms FAIL (2025-10-27 13:01:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 198.315322ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................XX...X......................] duration: 384.426075ms FAIL (2025-10-27 13:00:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 384.462841ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 189.55667ms FAIL (2025-10-27 13:00:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 189.594453ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 198.518258ms FAIL (2025-10-27 12:59:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 198.551226ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................XX...X.....................] duration: 394.59011ms FAIL (2025-10-27 12:59:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 394.622404ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................XX...X....................] duration: 423.118607ms FAIL (2025-10-27 12:59:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 423.150493ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 242.557785ms FAIL (2025-10-27 12:58:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 242.612645ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................XX...X...................] duration: 395.646102ms FAIL (2025-10-27 12:58:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:45:38.667404919 (733h27m24.577776776s) [upload=762248]: Starting upload Oct 27 12:45:38.667464159 (733h27m24.577836032s) [upload=762248]: Done starting upload Oct 27 12:45:38.667807593 (733h27m24.578179453s) [upload=762248]: Uploading segments... Oct 27 12:45:38.667874772 (733h27m24.57824664s) [upload=762248]: Got next segment. Inline: false Oct 27 12:45:38.667915143 (733h27m24.578287s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:45:38.667922654 (733h27m24.578294522s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 12:45:38.706331218 (733h27m24.616703092s) [seg_pos={0 0}, upload=762248]: Begin upload segment... Oct 27 12:45:38.706407791 (733h27m24.616779661s) [seg_pos={0 0}, upload=762248]: Done begin upload segment. Oct 27 12:45:38.706453462 (733h27m24.616825336s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706472018 (733h27m24.616843893s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706536403 (733h27m24.616908272s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706959467 (733h27m24.617331335s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.706981918 (733h27m24.61735379s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707474367 (733h27m24.617846244s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707523876 (733h27m24.617895746s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707904560 (733h27m24.618276428s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.707924689 (733h27m24.61829655s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708228407 (733h27m24.618600275s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708550551 (733h27m24.618922419s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.708859181 (733h27m24.61923105s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709390661 (733h27m24.619762533s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709543805 (733h27m24.619915673s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.709765947 (733h27m24.620137818s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710060610 (733h27m24.620432464s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710181440 (733h27m24.620553309s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.710793230 (733h27m24.621165101s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Uploading piece... Oct 27 12:45:38.783197983 (733h27m24.693569844s) [upload=762248]: Next returned nil segment Oct 27 12:45:38.783214763 (733h27m24.693586635s) [upload=762248]: Waiting for error group managing segments... Oct 27 12:45:38.787755505 (733h27m24.698127377s) [noise=true , address=45.91.116.6:4977 , piece_id=GDJUOLNXRGPC6PEMNYG6D76SKUZTLK5DSKXLY5RSJTKS4P7C6PSQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789173882 (733h27m24.699545742s) [noise=true , address=77.90.31.15:4706 , piece_id=OA5BDXNRX4RZRDT7SEDT76F3IW4G344VWUWA3ITIAP4PIDF75DTA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.789639188 (733h27m24.700011043s) [noise=true , address=145.79.227.2:4125 , piece_id=26FVAKMSXLSWOQHBM4X55OJTZRWPYUBIRXYUROFETS3UL3VCLCIQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.791988865 (733h27m24.702360727s) [noise=true , address=77.90.31.12:4610 , piece_id=D4YBSL2CCQQAFWQ6RQ4PIASQMRS6ETAMKPQ6HRR4PO3O6TXCWEYA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.797477828 (733h27m24.707849702s) [noise=true , address=77.90.31.14:4669 , piece_id=577LM2MZ3JYQO5FGVCXJJMHTN2SUTYXKKINJYRCNAKMTYKMCDIPQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.800309737 (733h27m24.710681598s) [noise=true , address=45.91.116.5:4301 , piece_id=GMWSXGZOESJ4P5L5JBJZKNRHVQ4OUGYWRPFRYQ4HZHPFGCEYOUWA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.801359604 (733h27m24.711731465s) [noise=true , address=145.79.227.46:4613 , piece_id=LI6DRYFIELUXXAZGCIUZBP5BNR4T7ITQQ5IFFK5OOOBIW33YPFNA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802468760 (733h27m24.712840622s) [noise=true , address=45.91.116.4:4948 , piece_id=6HGRJUJKHLUUU5IOYIVZUT74MMSQJMT4FJOTF3BLTAMJN62FS5BA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.802910032 (733h27m24.713281893s) [noise=true , address=45.91.116.8:4065 , piece_id=OUXWBNQGWQYGQCKMEDR435TPYDWCYHLBV5IT4CGFUCZSWN6FYA3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804479635 (733h27m24.714851493s) [noise=true , address=45.91.116.12:4247 , piece_id=CSZQ3RMYJEJCZMTJDYEJUZUYVK47EO7EUYOQOULNMXH7VUL6GJKQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.804931711 (733h27m24.715303569s) [noise=true , address=145.79.227.19:4099 , piece_id=XDB6WWFJ3GFRHZERYOCHHNSNODWKETP5FAM43GRRP6T6BHDIPNMA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806183322 (733h27m24.716555184s) [noise=true , address=77.90.31.16:4753 , piece_id=22DO6I6YEX2A5DRDBPYGSX4IIF3FMQRZKSR5CTQE5NQ3TQDYOCTQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.806467094 (733h27m24.716838964s) [noise=true , address=45.91.116.7:4023 , piece_id=UH7LYWBURRCV7OIMS34NTSHXP2XLN47GN3RHTULZ4T2ZAWWQUV3Q, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832791432 (733h27m24.743163302s) [noise=true , address=77.90.31.13:4629 , piece_id=RRPUZNWUT7B2UVTFB7WOHVUERP3SMMDSP4CCJDZOF45HISC4WYCA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.832825975 (733h27m24.743197844s) [seg_pos={0 0}, upload=762248]: Segment reached optimal threshold of 14 pieces. Oct 27 12:45:38.833044165 (733h27m24.743416025s) [noise=true , address=77.90.31.18:4830 , piece_id=O6GPFF3RBOFYKF6EW7NXJAAEVSAGBVYKAGKLNWWWXVKLZBR7OVKA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: <nil> Oct 27 12:45:38.833062873 (733h27m24.743434742s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833077136 (733h27m24.743449014s) [noise=true , address=77.90.31.21:4880 , piece_id=LQF6A7PEIY2R2WO7H2AULRTO5JYO7EKUWSBDO4CJ2M4SJ3NKF3AQ, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833182567 (733h27m24.743554424s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE): protocol: expected piece hash; context canceled; EOF Oct 27 12:45:38.833190367 (733h27m24.743562238s) [noise=true , address=77.90.31.10:4544 , piece_id=ANTAVDRNSXKTZIIQQ3HPHR66NVUKLHMDHNNNSIJXN4GK25QSDWIA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833417833 (733h27m24.743789703s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: read tcp 159.223.143.39:37978->64.49.13.170:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 27 12:45:38.833433085 (733h27m24.743804964s) [noise=true , address=64.49.13.170:4722 , piece_id=ON6JXIWAAF2JHASFZMG22TEKENJDUHU7FMVFOLBA3FPQBDS6EHRA, seg_pos={0 0}, upload=762248]: Long tail cancellation due to optimal threshold Oct 27 12:45:38.833460377 (733h27m24.743832246s) [seg_pos={0 0}, upload=762248]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Oct 27 12:45:38.833546717 (733h27m24.743918586s) [upload=762248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 12:45:38.833552788 (733h27m24.743924657s) [upload=762248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:46:38.652657858 (733h28m24.563029718s) [upload=762248]: Done uploading segments... Oct 27 12:46:38.652686704 (733h28m24.563058572s) [upload=762248]: Upload finished. err: metaclient: context deadline exceeded Duration: 395.678442ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 245.149419ms FAIL (2025-10-27 12:58:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 245.17368ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 232.89127ms FAIL (2025-10-27 12:57:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 27 12:42:07.837922634 (733h23m53.748294507s) [upload=762189]: Starting upload Oct 27 12:42:07.838022900 (733h23m53.748394773s) [upload=762189]: Done starting upload Oct 27 12:42:07.838119945 (733h23m53.748491814s) [upload=762189]: Uploading segments... Oct 27 12:42:07.838198500 (733h23m53.74857038s) [upload=762189]: Got next segment. Inline: true Oct 27 12:42:07.838232452 (733h23m53.748604321s) [upload=762189]: Waiting for error group managing segments... Oct 27 12:42:07.838289422 (733h23m53.7486613s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 12:42:07.838312024 (733h23m53.748683893s) [upload=762189]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 27 12:42:07.838316458 (733h23m53.748688327s) [upload=762189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 12:43:07.797330135 (733h24m53.707702006s) [upload=762189]: Done uploading segments... Oct 27 12:43:07.797362585 (733h24m53.707734463s) [upload=762189]: Upload finished. err: metaclient: context deadline exceeded Duration:* 232.932314ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)